Dynamic routing based on namespace

[thenu97]

A note for the community

• Please vote on this issue by adding a 👍 reaction to the original issue to help the community and maintainers prioritize this request
• If you are interested in working on this issue or have submitted a pull request, please leave a comment

Use Cases

We currently have an EKS cluster where each client has their own namespace. We want to be able to send each client logs to an EC2 instance that belongs to the client.

Attempted Solutions

We tried deploying a vector instance per namespace, however, we quickly realised that having 300 namespaces, it'll be a vector per node per namespace so that'll be 300 pods per node, which obviously isn't ideal.

Proposal

We're hoping we will be able to do some intelligent namespace based routing e.g.

data_dir: /vector-data-dir
sources:
  kube_logs:
    type: kubernetes_logs
    auto_partial_merge: true
    # extra_field_selector: "metadata.namespace=${namespace}"

transforms:
  add_metadata:
    inputs:
      - kube_logs
    type: remap
    source: |
      .namespace = .kubernetes.namespace
      .cluster_name = "${cluster_name}"

sinks:
  namespace_router:
    type: vector
    inputs:
      - add_metadata
    address: "{{ namespace }}-0.hosting.test.com:3150"

however type: vector doesn't have address template-able.

If this isn't the best way of handling, please let us know what the best practice is in such cases

References

No response

Version

0.40.1

[pront]

Hi @thenu97, thanks for creating this. I cannot think of a workaround that works elegantly for a large number of namespaces.

This is a nice feature but it will require a number of changes in the Vector sink.