Skip to content

Commit

Permalink
Fix race condition in PageListener
Browse files Browse the repository at this point in the history
This PR
- Introduced an `AtomicInteger` called `pagesInFlight` to track the number of pages currently being processed. 
- Incremented `pagesInFlight` before processing each page and decremented it after processing is complete
- Adjusted the condition in `scheduleImputeHCTask` to check both `pagesInFlight.get() == 0` (all pages have been processed) and `sentOutPages.get() == receivedPages.get()` (all responses have been received) before scheduling the `imputeHC` task. 
- Removed the previous final check in `onResponse` that decided when to schedule `imputeHC`, relying instead on the updated counters for accurate synchronization.

These changes address the race condition where `sentOutPages` might not have been incremented in time before checking whether to schedule the `imputeHC` task. By accurately tracking the number of in-flight pages and sent pages, we ensure that `imputeHC` is executed only after all pages have been fully processed and all responses have been received.

Testing done:
1. Reproduced the race condition by starting two detectors with imputation. This causes an out of order illegal argument exception from RCF due to this race condition. Also verified the change fixed the problem.
2. added an IT for the above scenario.

Signed-off-by: Kaituo Li <[email protected]>
  • Loading branch information
kaituo committed Dec 3, 2024
1 parent 1a3b8c9 commit 4661ada
Showing 1 changed file with 3 additions and 0 deletions.
Original file line number Diff line number Diff line change
Expand Up @@ -239,6 +239,9 @@ public void onResponse(CompositeRetriever.Page entityFeatures) {
scheduleImputeHCTask();
}

// Increment pagesInFlight to track the processing of this page
pagesInFlight.incrementAndGet();

if (entityFeatures != null && false == entityFeatures.isEmpty()) {
LOG
.info(
Expand Down

0 comments on commit 4661ada

Please sign in to comment.