Allow setting ingest pipeline via filebeat hints

[segevfiner]

Describe the enhancement:
You can already set all/most other properties via autodiscover hints, but it seems that setting an ingest pipeline is missing. Although you can set a module/fileset which do use ingress pipelines internally, working around this using modules when it isn't really made convenient to create and deploy third party out of tree modules. So this is about setting a custom ingress pipeline, created via other means, via autodiscover hints, behaving similarly to the input/output properties available in the configuration file.

Describe a specific use case for the enhancement or feature:

Being able to create a custom ingest pipeline directly in ES and set filebeat to use it for a specific container/pod via autodiscover hints without having to modify filebeat configuration and invent some custom label/annotation to make the pipeline apply only to that container/pod.

[elasticmachine]

Pinging @elastic/integrations-platforms (Team:Platforms)

[babadofar]

If you really mean "ingest pipeline" and not "ingress pipeline", I was just looking for the same thing. I actually assumed it already was possible. Specifying pipelines pr. applicaiton makes a lot of sense.

[PBarnard]

If you really mean "ingest pipeline" and not "ingress pipeline", I was just looking for the same thing. I actually assumed it already was possible. Specifying pipelines pr. applicaiton makes a lot of sense.

Agreed. This would be really helpful. I'm currently having to feed all log events into a single Ingest Pipeline then from there using the "pipeline" processor to branch off depending on Kubernetes namespace or application name.

[botelastic]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

[PBarnard]

@elastic/integrations-platforms any update on this?

[ChrsMark]

Hi @PBarnard and @segevfiner!

We will need to evaluate this further however to be honest I find this addition quite tricky since it would open a significant security discussion. With a pipeline you can load actually a "script" in ES which could do anything. So imagine that one Pod could be deployed with this hint and the pipeline could manipulate events coming into ES from whatever other resource. In addition it would be quite tricky to set propel permissions on this so I would avoid doing this.

Usually ingest pipelines are coming along with modules. In that case did you consider creating a complete module with the required pipeline inside it?

cc: @mukeshelastic @akshay-saraswat @MichaelKatsoulis @tetianakravchenko

[segevfiner]

Considering you already allow setting modules and processors how would setting the name of an existing ingress pipeline would be any different from a security standpoint?

(Yeah, defining an entire ingress pipeline in an annotation could be risky, but that's not what I'm asking for).

Last I checked modules weren't very convenient to create outside using tools in the filebeat source tree and with lacking documentation in some areas, though it has been a while since I last tried. (This issue was opened in 2020, it is 2022 now)

[ChrsMark]

Hey @segevfiner thanks for elaborating on this. Initially I thought about providing the option to define and set the entire pipeline.

Maybe setting the just the pipelineId would be a nice to have feature. See https://www.elastic.co/guide/en/beats/filebeat/current/filebeat-input-log.html#_pipeline_12.

I will bring it to team's planning and we should be evaluating this soon!