Feature: Per namespace elastic + filebeat?

[KlavsKlavsen]

Hi guys,

Loving this ECK.. I would like to setup an elastic instance per-namespace (as we give each team their own namespace) - so they can have their own setup - isolated from other teams.

That works fine for elastic+kibana..
But the filebeat example you have - reads in from ALL containers on host.. it would be really nice if there could be some way to have a single filebeat instance (as your example yaml sets up) - which then can filter based on which namespace the pod belongs to - and direct logs to the correct elastic instance)

To avoid teams logs bothering each other - and avoiding all the complexity (and possible flaws) - in having a central elastic search with filtering for each team - so they only see their own nodes..

[david-kow]

Hi @KlavsKlavsen, thanks for your feature request.

Since a single Filebeat can't be configured to have multiple outputs this is not easy to set up. I didn't have an opportunity to test the below, but two options I can think of right now is to either:

• Deploy Filebeat as a sidecar in each Pod and either mount the appropriate log file from the host or write to file inside the Pod.
• Deploy one Filebeat DaemonSet per namespace and use autodiscovery to pin each DeamonSet to a single namespace.

We are currently exploring if this kind of scenario is something that ECK could help solve.

[KlavsKlavsen]

Thanks @david-kow
My current plan is to setup 1 instance of filebeat to pull all logs and push to a kafka (for sysadmins only).. it would be nice if kafka output supported splitting to a topic per namespace f.ex..

And then I'll set up 1 instance of filebeat per team namespace - where I hope that I can filter on a hardcoded namespace name - so the ES in that namespace (which will be the output for this filebeat) - only will get logs from containers in that teams namespace.

[anyasabo]

Closing as this will be tracked in #2417

[kfox1111]

Is that the right url? I can't seem to see it. 404. Permission issue?

[anyasabo]

Woops, I put in the wrong link. Corrected