Invoke default pipeline of new index #85931
Conversation
felixbarny
added
:Data Management/Ingest Node
elasticmachine
added
the
Team:Data Management
|
Pinging @elastic/es-data-management (Team:Data Management) |
elasticsearchmachine
added
the
external-contributor
|
Hi @felixbarny, I've created a changelog YAML for you. |
|
@felixbarny is there anything where I can help ? |
|
@rahuldimri thanks for asking. We're still in internal discussion on how to solve #63798. This PR is part of a prototype which may not make it in as-is. Can I ask you why you're interested in this PR? Do you have a use case that could be solved by this? If so, what's your use case? |
felixbarny
force-pushed
the
invoke-default-pipeline-of-new-index
branch
from
323c923 to
539d6ad
Compare
| @@ -39,6 +39,7 @@ | |||
| */ | |||
| public class Metadata { | |||
| protected static final String INDEX = "_index"; | |||
| protected static final String REDIRECT = "_redirect"; | |||
There was a problem hiding this comment.
I'm unsure if _redirect should also be added to IndexRequest. I guess that could potentially expose _redirect to the _bulk API where this doesn't make much sense. It's also more like an alias to _index but with slightly different semantics.
@dakrone do you have a suggestion?
| executePipelines(pipelines.iterator(), hasFinalPipeline, indexRequest, ingestDocument, documentListener); | ||
|
|
||
| LinkedHashSet<String> indexRecursionDetection = new LinkedHashSet<>(); | ||
| indexRecursionDetection.add(indexRequest.index()); |
There was a problem hiding this comment.
More general question: I assume during the routing, index() will always be the data_stream name for data_streams and the data_stream is only resolved to a write index after the processing happened? Asking it here because the data stream name is the one that should be part of the index recursion detection, not the write index as this could even change during ingest time I assume.
There was a problem hiding this comment.
Yes, _index isn't the internal write index of the data stream.
|
Closing in favor of #94000 |
When a processor changes the
_indexfield, only the final pipeline of the new index will be executed.This change preserves that behavior but introduces another
_redirectmetadata field that behaves similarly to setting_index.The difference is when setting this field, the default pipeline of the target index will be executed. This PR also detects and prevents loops in the index redirection.
Example: both index_1 and index_2 have a default and final pipeline. the default pipeline of index_1 sets the _redirect field to
index_2. This executes the pipelines index_1.default, index_2.default, index_2.final.Required for
Previous discussions: