Increase use of Go templates in parameters

[lovromazgon]

It's getting more and more common that we need to add support for Go templates in a specific processor or connector parameter. The usual use case is that some data from the record needs to be used to inject it into a parameter. The latest use case was taking data from the record when setting a header in the webhook.http processor.

This discussion is about leaning heavily into Go templates and using them to provide a way to inject record data where needed. We can take inspiration from GoReleaser, which offers similar functionality. For instance, in their documentation they document fields that support Go templates with Templates: allowed. Since all fields work pretty much the same way, that's all the documentation it needs regarding Go templates, while the behavior and available fields are described in a single place.

If we decide to go down this path it will be required to add utilities around Go templates for connector and processor developers to make the usage of Go templates easy, as well as to ensure the same user experience everywhere. For instance, it would be great if a config field could have the type template.Template and the config parsing utility would take care of parsing the template. Or maybe we could provide a wrapper around template.Template which ensures that you have to supply an opencdc.Record to execute the template.

Another thing we could address by providing such a utility is to make addressing record fields case insensitive. Right now, users have to access opencdc.Record fields using uppercase names like .Key, because that's the name of the field in the Go type. This can be confusing, especially for non-Go users, as the JSON representation of the record contains lower-case field names, which doesn't match the usage in Go templates. Our utility could parse the Go template, find any lower-case references to record fields, and replace them with the correct form (e.g. .key would become .Key).

The downside of this is worse performance since the template is executed for each record, although we could lower the impact by skipping the evaluation if we detect that the Go template contains a static string.

[hariso]

I agree with using a template type for fields that can be a Go template. It's likely the only solution to this problem. Also, we can then easily figure out where templates are allowed and where not.