Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Fix bug where retries within RemoteStoreRefreshListener cause infos/checkpoint mismatch #10655

Merged
merged 2 commits into from
Oct 19, 2023

Commits on Oct 19, 2023

  1. Fix bug where retries within RemoteStoreRefreshListener cause mismatc…

    …h between ReplicationCheckpoint and uploaded SegmentInfos.
    
    Retries within RemoteStoreRefreshListener run outside of the refresh thread.  This means that concurrent refreshes
    may occur during syncSegments execution updating the on-reader SegmentInfos.  A shard's latest ReplicationCheckpoint
    is computed and set in a refresh listener, but it is not guaranteed the listener has run before the retry fetches the infos or checkpoint independently.
    This fix ensures the listener recomputes the checkpoint while fetching the SegmentInfos. This change also
    ensures that we only recompute the checkpoint when necessary because it comes with an IO cost to compute StoreFileMetadata.
    
    Signed-off-by: Marc Handalian <[email protected]>
    
    Update refresh listener to recompute checkpoint from latest infos snapshot.
    
    Signed-off-by: Marc Handalian <[email protected]>
    
    Fix broken test case by comparing segments gen
    
    Signed-off-by: Marc Handalian <[email protected]>
    
    spotless
    
    Signed-off-by: Marc Handalian <[email protected]>
    
    Fix RemoteStoreRefreshListener tests
    
    Signed-off-by: Marc Handalian <[email protected]>
    mch2 committed Oct 19, 2023
    Configuration menu
    Copy the full SHA
    0ce0d9d View commit details
    Browse the repository at this point in the history
  2. add extra log

    Signed-off-by: Marc Handalian <[email protected]>
    mch2 committed Oct 19, 2023
    Configuration menu
    Copy the full SHA
    04180e3 View commit details
    Browse the repository at this point in the history