add lua script to optionally remap metadata keys in kube-logs

[kruftik]

Fixes fluent/fluent-bit#1134 by injecting a lua function from fluent/fluent-bit#1134 (comment) to daemonset pods and using it to replace dots in the keys of kubernetes labels and annotations with underscore to fix data type-related conflicts

[naseemkullah]

Hi @kruftik, have you considered using the Replace_Dots option for the es output?

[kruftik]

this option is available only in the case of direct shipping to ES instances, however there are plenty of other options. for instance, shipping to logstash node in front of ES cluster

[naseemkullah]

Is this a problem you are facing today? logstash has https://www.elastic.co/guide/en/logstash/current/plugins-filters-de_dot.html if so.

That aside, as per fluent/fluent-bit#1134 (comment) it has been suggested to open a pr to add this to default scri=ipts of fluent bit, which would be better than putting it in the chart as to keep the chart lightweight.

With that said, the script is not included yet (has a pr been opened yet?), lets add instructions in readme for generl purpose lua script usage with this as an example. WDYT?

[Cyanopus]

Can we have this functionality merged as its literally impairing all other setups apart from fluent-bit to ES?We are running fluentbit to fluentd as recommended everywhere and we don't want to put yet one more filter on the aggregator part to accommodate such a small change.

[naseemkullah]

the idea behind this chart is to be lightweight and highly flexible, and not hard code such use cases, though if this is a script that is in high enough demand it should be added to fluent-bit as per fluent/fluent-bit#1134 (comment), please submit a pr there if this is something you see as mandatory out of the box. Else, this chart could make do with a more general approach of mounting lua scripts in general, today this can be achieved with extraVolume/extraVolumeMounts if I'm not mistaken.

[kruftik]

@naseemkullah ,

does the chart need a general way to easy mounting some auxiliary Lua scripts) into the fluent-bit PODs for some advanced log processing or such stuff is out of chart scope at all?

[naseemkullah]

@naseemkullah ,

does the chart need a general way to easy mounting some auxiliary Lua scripts) into the fluent-bit PODs for some advanced log processing or such stuff is out of chart scope at all?

Hi @kruftik, imho a generic way to create/consume lua scripts would be a good feature, or a documented process as how this can be achieved today (i.e. first create a configmap with the script, attach it as a volume, and so on).

[kruftik]

ok, we'll prepare a PR with such a feature soon, close this one.