Allow table and database config options in Clickhouse sink to be templatable

[Fritzss]

A note for the community

• Please vote on this issue by adding a 👍 reaction to the original issue to help the community and maintainers prioritize this request
• If you are interested in working on this issue or have submitted a pull request, please leave a comment

Use Cases

scenario:
server A sends data to a vector on server B
server B receives and sends to clickhouse
(server A -> n, there can be many)
################
src svc_logs_server cat /etc/vector/logs.toml

data_dir = "/var/lib/vector"

[acknowledgements]
enabled = true

[sources.logs_svc]
type = "file"
file_key = "file"
glob_minimum_cooldown_ms = 1_000
host_key = "host"
ignore_older_secs = 600
include = [
            "/var/log/svc/*.log"
          ]

max_line_bytes = 1024000
max_read_bytes = 1048576
read_from = "beginning"
data_dir = "/var/lib/vector"

[transforms.json_svc]
type = "remap"
inputs = ["logs_svc"]
source = """
        . = merge(object!(parse_json!(.message, max_depth: 1)), . )
        .table = del(.message)
        .table = "svc_logs"
         """
[sinks.vector_ch]
  type = "vector"
  inputs = [ "json_svc" ]
  address = "10.0.0.123:6000"
  compression = true
  buffer.type = "memory"
  buffer.max_events = 5000
  buffer.when_full = "block"
  batch.max_events = 1000
  batch.timeout_secs = 60
  request.concurrency = "adaptive"

dst_server clickhouse + vector cat /etc/vector/vector.toml

data_dir = "/var/lib/vector"


[acknowledgements]
enabled = true


# Random Syslog-formatted logs
[sources.vector]
  type = "vector"
  version = "2"
  address = "0.0.0.0:6000"


# Print parsed logs to stdout
#[sinks.print]
# type = "console"
# inputs = ["vector"]
# encoding.codec = "json"

[sinks.clickhouse_vault]
  type = "clickhouse"
  inputs = [ "vector" ]
  endpoint = "http://127.0.0.1:8123"
  database = "logs"
  table = {{ .table }}
  encoding.timestamp_format = "unix"
  buffer.type = "memory"
  buffer.max_events = 5000
  buffer.when_full = "block"
  batch.max_events = 1000
  batch.timeout_secs = 20
  skip_unknown_fields = true
  request.concurrency = "adaptive"

Attempted Solutions

there is not enough possibility to declare a global variable in the config
GLOBAL_VAR in config.toml

Proposal

No response

References

• How to use fields in clickhouse sincs (table, database ?) #16327
• Rewrite the clickhouse sink into the new style #17094

Version

vector 0.26.0

[r3code]

I'm trying the same. I want to make the sink cobfig dynamic. Based on message fields select clickhouse database and table. I planned to add a transform before clickhouse sink which adds internal fields like ._clickhouse_db and ._clickhouse_table, then use it in sink as a templates value.

[r3code]

Why we need this? This allows us to build vector config once and not adding new transforms for new services every time we need to enable logs for a new service.
All that I need is to have my clickhouse tables created.