-
Notifications
You must be signed in to change notification settings - Fork 24.9k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Applying destination index pipelines after a processor changes the target index #57968
Comments
Pinging @elastic/es-core-features (:Core/Features/Ingest) |
CC: @ruflin |
I think we should aim to keep logic of executing pipelines after the _index has been changed simple. Prior to performing ingest, a list of pipelines is determined. These pipelines are executed. If
Given with the above logic, yes.
I think only the final pipeline should be executed.
I think it is fine if request or default pipelines change the index. However I think that final pipelines shouldn't be changing the index. |
We discussed this issue with a broader audience and think that the re-execution logic of pipeline should be limited:
The above restrictions would avoid infinitely redirections between pipelines. |
Just to check: In the current proposal, if a default pipeline redirects to another index then the default pipeline on that new destination index would not be executed. Is that correct? I do wonder whether that works for Ingest Manager - My understanding is that they use both default and final pipelines, and would want to rely on both being executed after routing to a new index. |
Yes, that is the idea.
@ruflin Can you comment on whether the proposal works for ingest manager? |
@martijnvg Looking at your proposal I think this makes sense. @andresrc and @exekias do this still work in the routing idea? I think so. |
for ingest manager, since we control the pipelines, we could reasonably make the necessary adjustments to the final destination "final" pipelines to make this work. Sketching the solution out:
One downside that comes to mind is that if a user wants to bypass the "final" pipeline for some reason, they can't. As an example, as far as I understand, reindexing will apply the "final" pipeline again, which could be problematic. I don't want to derail the conversation too much, but I think it's worth considering if we need to take a step back and think about building the right abstractions into Elasticsearch for a routing/parsing pipeline like this instead of making do with what we have - similar to what we did with data streams - which I think will end up being incredibly valuable for our users over the long term because we now have an abstraction that fits much better with what our users are trying to achieve and it paves the way for us to deliver new features on this abstraction that would have been otherwise very difficult. |
The idea of final pipelines is that it can't be bypassed, so maybe in that case a default pipeline should be used?
👍 I think it is a good idea to take a good look at the current implementation and consider alternatives for it. |
This change fixes final pipelines if destination index is changed during pipeline run: -final pipelines can't change destination anymore, exception is thrown if they try to -if request/default pipeline changes destination final pipeline from old index won't be executed -if request/default pipeline changes destination and new index has final pipeline it will be executed -default pipeline from new index won't be executed Additionally TransportBulkAction.resolvePipelines was moved to IngestService as it's needed for resolving pipelines from new index. Tests were moved accordingly. Closes #57968
…59522) This change fixes final pipelines if destination index is changed during pipeline run: -final pipelines can't change destination anymore, exception is thrown if they try to -if request/default pipeline changes destination final pipeline from old index won't be executed -if request/default pipeline changes destination and new index has final pipeline it will be executed -default pipeline from new index won't be executed Additionally TransportBulkAction.resolvePipelines was moved to IngestService as it's needed for resolving pipelines from new index. Tests were moved accordingly. Closes elastic#57968
…9522) (#59746) * Fix handling of final pipelines when destination is changed (#59522) This change fixes final pipelines if destination index is changed during pipeline run: -final pipelines can't change destination anymore, exception is thrown if they try to -if request/default pipeline changes destination final pipeline from old index won't be executed -if request/default pipeline changes destination and new index has final pipeline it will be executed -default pipeline from new index won't be executed Additionally TransportBulkAction.resolvePipelines was moved to IngestService as it's needed for resolving pipelines from new index. Tests were moved accordingly. Closes #57968
…astic#59522) (elastic#59746) * Fix handling of final pipelines when destination is changed (elastic#59522) This change fixes final pipelines if destination index is changed during pipeline run: -final pipelines can't change destination anymore, exception is thrown if they try to -if request/default pipeline changes destination final pipeline from old index won't be executed -if request/default pipeline changes destination and new index has final pipeline it will be executed -default pipeline from new index won't be executed Additionally TransportBulkAction.resolvePipelines was moved to IngestService as it's needed for resolving pipelines from new index. Tests were moved accordingly. Closes elastic#57968
…9522) (#59756) * Fix handling of final pipelines when destination is changed (#59522) This change fixes final pipelines if destination index is changed during pipeline run: -final pipelines can't change destination anymore, exception is thrown if they try to -if request/default pipeline changes destination final pipeline from old index won't be executed -if request/default pipeline changes destination and new index has final pipeline it will be executed -default pipeline from new index won't be executed Additionally TransportBulkAction.resolvePipelines was moved to IngestService as it's needed for resolving pipelines from new index. Tests were moved accordingly. Closes #57968
When evaluating ingest pipelines, we determine the full set of pipelines upfront (taking into account default_pipeline and final_pipeline) based on the index name that is included in the incoming request.
While executing those pipelines it is possible that a processor will change the destination index , for example (trivial example)
If that happens it is possible that the new destination index has its own set of default/final pipelines, but we do not execute them.
It would be preferable if the pipelines attached to the new destination were executed, although there are some questions/issues to work through
The text was updated successfully, but these errors were encountered: