[Agent] Add templating support inside the Agent configuration.

[ph]

In the current modules today we allow users to reference variable that will be replaced when the modules are loaded into Filebeat. Like this:

var:
  - name: paths
    default:
      - /var/log/mysql/error.log*
      - /var/log/mysqld.log*
    os.darwin:
      - /usr/local/var/mysql/{{.builtin.hostname}}.{{.builtin.domain}}.err*
    os.windows:
      - "c:/programdata/MySQL/MySQL Server*/error.log*"

In the above example, we have two placeholder values.

• {{.builtin.hostname}}
• {{.builtin.domain}}

In issue #279 we discussed where this replacement could happen and we have decided that it make sense to allow it to be executed in the Agent with a few hard rules in place.

• Define a limited set of accessible values.
• Share this set of values with the package-registry to do some package validation.
• If the we try to reference invalid or non-existing values we should fail hard and early in the validation chain. (before sending it to the process)

Possible Values

We should keep the list of exposed values short and follow ECS naming here. Here as an early list of values we could expose.

• Version of the Agent.
• Platform (Windows, Darwin, linux)
• Architecture.
• Hostname
• Domain

Syntax

The syntax used in the current module definition is the one from the golang template syntax. This means the YAML is read as a Golang template and parsed again as a ucfg configuration. I think we should not go that route:

• Because the golang templates brings more than just values (loops, conditions, etc)
• This could allow users to changes the YAML keys using templates snippets.

Instead, I see two possible options:

1. Using the current field reference syntax.

We could use a superset of the field reference syntax in beats, this would mean that we have to implement a custom resolver from go-ucfg. This will effectively limit to only replacing values in the YAML values and nothing else.

${.agent.version}
${.agent.host}
or
${var:agent.host}

Usage:

paths:
      - /usr/local/var/mysql/${.agent.hostname}.${.agent.domain}}.err*

2. Use a handlebar-like library

Implements or use a golang's Handlebar library with only support variable replacement and no logical operator. From what I see, the majority of the template engine defines more than just values replacement.

Where this is used?

Currently, the only module I see that uses it is the MySQL module in filebeat.

winterfell~/go/src/github.com/elastic/beats(master|✔) % ag 'builtin' --yaml
filebeat/module/mysql/slowlog/manifest.yml
7:      - /var/lib/mysql/{{.builtin.hostname}}-slow.log
9:      - /usr/local/var/mysql/{{.builtin.hostname}}-slow.log*

filebeat/module/mysql/error/manifest.yml
9:      - /usr/local/var/mysql/{{.builtin.hostname}}.{{.builtin.domain}}.err*

Required decisions

• List of fields
• Corresponding ECS field.
• Syntax to use for referencing the values.

[elasticmachine]

Pinging @elastic/ingest-management (Team:Ingest Management)

[ph]

@ruflin @michalpristas I think option 1 above is the easiest for the Agent to implement, we just have to implement: a store of values and a resolver for go-ucfg. This is the exact same strategy I've used in the keystore which worked fine. This will also guarantee that only values replacement is available. Also, we could implement it easily on top of our Tree package in the Agent as a visitor.

The downside is if we need similar behavior on the Kibana side we would have to have a custom parser. But still, theses are just values replacement and this should not be too hard to add.

[ruflin]

From a package perspective I would prefer to only have 1 template language, otherwise it might become confusing for a package dev on what syntax to use where.

One thing I realised reading the issue is that it is important all the fields we use are constant and global. They don't depend on a specific event or the way the Beat was started. There might be chance that some of the values like hostname might change during running the agent but it is rare.

I would say we delay the detailed discussion on what template language should be used until we start the implementation.

[ph]

One thing I realised reading the issue is that it is important all the fields we use are constant and global. They don't depend on a specific event or the way the Beat was started. There might be chance that some of the values like hostname might change during running the agent but it is rare.

Correct hostname could change,

I would say we delay the detailed discussion on what template language should be used until we start the implementation.

Agree on that, there is only one module that uses it. We can delay that specific package too.

[ruflin]

Closing this in favor of #19225