@Custom Pipeline Behaviour

[WildDogOne]

Hello Everyone,

Since (I think) V8.4 every integration pipeline automatically gets an appended @Custom pipeline
This of course makes a lot of sense, up till the point where we have an integration with chained Pipelines

Example Apache:
Pipeline "logs-apache.access-x.x.x" triggers "logs-apache.access-x.x.x-third-party"
and both of them trigger "logs-apache.access@custom"

This is redundant, and in worst case could bring about strange behaviour and errors.
I have no idea if this issue has been reported yet, and I don't even know if the integrations section is the right place...

Regards,

linus

[andrewkroh]

@nchaulet Could you take a look at this. It looks like a bug. It sounds like the @custom is being injected into all pipelines instead of just the default/main pipeline for the data stream. Several integrations organize their logic using "sub-pipelines" where the main pipeline calls into a another pipeline based on some branching condition. We would only want the @custom to be injected into the a main pipeline.

Relates: elastic/kibana#134578

[elasticmachine]

Pinging @elastic/fleet (Team:Fleet)

[nchaulet]

@nchaulet Could you take a look at this. It looks like a bug. It sounds like the @Custom is being injected into all pipelines instead of just the default/main pipeline for the data stream. Several integrations organize their logic using "sub-pipelines" where the main pipeline calls into a another pipeline based on some branching condition. We would only want the @Custom to be injected into the a main pipeline.

Yes you are correct looking at the code we inject the @custom pipeline for all the pipelines and not only the main one, it seems an easy one to fix.