Reindex resolve indices early #49850
Closed
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Resolve indices before starting to reindex. This ensures that the list of indices does not change when failing over (TBD). The one exception to this is aliases, which we still need to access through the alias. In addition, resolved index patterns are sorted by create-date and otherwise the listed order is preserved. This ensures that once we reindex one index at a time, we will get reasonable time locality for time based indices. The resolved list of indices will also by used to do searching one index (or index group) at a time, improving search performance (since we use sort) and allowing us to do more fine-grained checkpoint and track progress (TBD). Relates elastic#42612
henningandersen
added
>non-issue
:Distributed Indexing/Reindex
|
Pinging @elastic/es-distributed (:Distributed/Reindex) |
|
ci/1 test failure fixed here: #49855 |
Tim-Brooks
reviewed
Dec 10, 2019
| @@ -87,7 +103,7 @@ protected void doExecute(Task task, StartReindexTaskAction.Request request, Acti | |||
|
|
|||
| // In the current implementation, we only need to store task results if we do not wait for completion | |||
| boolean storeTaskResult = request.getWaitForCompletion() == false; | |||
| ReindexTaskParams job = new ReindexTaskParams(storeTaskResult, included); | |||
| ReindexTaskParams job = new ReindexTaskParams(storeTaskResult, resolveIndexPatterns(request.getReindexRequest()), included); | |||
There was a problem hiding this comment.
Do we want to be putting this in the cluster state? I don't have a sense for how large this gets, but I assume it could go in the index?
There was a problem hiding this comment.
Thanks, this is clearly wrong, fixed in 33d8fd4.
|
Marking this WIP and intend to close later based on our discussions the other day. The conclusion there was that we should try to improve this inside search rather than try to improve performance in reindex by doing one index at a time. |
rjernst
added
the
Team:Distributed (Obsolete)
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Resolve indices before starting to reindex. This ensures that the list
of indices does not change when failing over (TBD). The one exception to
this is aliases, which we still need to access through the alias.
In addition, resolved index patterns are sorted by create-date and
otherwise the listed order is preserved. This ensures that once we
reindex one index at a time, we will get reasonable time locality for
time based indices.
The resolved list of indices will also by used to do searching one
index (or index group) at a time, improving search performance (since we
use sort) and allowing us to do more fine-grained checkpoint and track
progress (TBD).
Relates #42612