How to leverage Flux to preserve the sequence of bulk puts... #1157
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
… the sequence of bulk puts in Document Migration to have a flux that effectively reports progress. Signed-off-by: Greg Schohn <[email protected]>
AndreKurait
reviewed
Nov 21, 2024
| @@ -76,7 +75,7 @@ Mono<Void> sendBulkRequest(UUID batchId, List<BulkDocSection> docsBatch, String | |||
| .log()) | |||
| // Prevent the error from stopping the entire stream, retries occurring within sendBulkRequest | |||
| .onErrorResume(e -> Mono.empty()) | |||
| .then() // Discard the response object | |||
| .then(Mono.just(new SegmentDocumentCursor())) // Discard the response object | |||
There was a problem hiding this comment.
note: this constructor would take in the last docsBatch
AndreKurait
reviewed
Nov 21, 2024
| @@ -307,6 +310,7 @@ public static void main(String[] args) throws Exception { | |||
| run( | |||
| LuceneDocumentsReader.getFactory(sourceResourceProvider), | |||
There was a problem hiding this comment.
note: we'll need to make the LuceneDocumentsReader read in order too
|
Thanks greg, remaining work tracked in #1160 |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Description
This is a partial and rough implementation of AndreKurait's idea that sequences bulk put responses into a flux that's effectively a progress report. We can then use that to monitor and update the last watermark that we've made it to for purposes of writing the work continuation record when a lease expires.
TODO:
Unit tests compile/pass
sendBulkRequest takes in the segment/docIds (at least the last one) & that's wired to the SegmentDocumentCursor
exitOnLeaseTimeout() is enhanced to do create a new work item from the lastDocIndexed
Make this easily exitOnLeaseTimeout() easily transferable to MIGRATIONS-2172
Category Enhancement
Why these changes are required? To support preserving backfill work even when a process is terminated.
What is the old behavior before changes and new behavior after changes? See Jira.
Testing
Tests break - this PR is to communicate the idea.
Check List
By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.