-
Notifications
You must be signed in to change notification settings - Fork 6
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Add BackfillStatus and Improve StartingVersion Selection on (re)start #10
Conversation
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
approved with a minor comment
.await | ||
.unwrap(); | ||
|
||
assert_eq!(starting_version, 11); | ||
} |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
nit: should we consider adding a test case where the configuration has a starting_version explicitly set?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
added! thanks
Purpose
Add new
backfill_status
Enum toBackfillProcessorStatus
. The enum can either beInProgress
orComplete
. This will help theget_starting_version
function decide whether or not to respect the latest checkpoint in the backfill processor status table or to just useconfig.starting_ver
/ genesis. Read more in the designTechnical Overview
Mostly the addition of logic into the
starting_version.rs
file. Otherwise, some convenience functions implementingToSql
andFromSql
for theBackfillStatus
enum.New logic will mark
BackfillStatus
asComplete
whenlst_success_version >= backfill_end_version
and asInProgress
otherwise.Testing
Added new dependency on the testing framework (!) which allows us to use a mock postgres instance so we can test the
starting_version
logic by manipulating the persisted state.Added 4 test cases to lock the intended behavior:
Complete
checkpointInProgress
checkpointPlease note I tried to reduce the setup code repetition in the tests but couldn't fix the connection refused error within my time box 🤷♂️
Also tested the above scenarios manually. Works as expected.