Add 5 min lookback for immutable streams to prevent missing records #45
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Description of change
We've observed a situation where events and balance_transactions can skip data when the record gets created exactly on (or very near) a date_window break. There appears to be some amount of time between when a record is stamped as created and when it's made available by the API. Likely only on the order of seconds.
For example
1574260351
.created[gte]=1574258564&created[lt]=1574260351&limit=100
The Fix
This PR is to add a 5 min lookback window to the start_date of all requests for
balance_transactions
andevents
. 5 minutes was chosen to account for potential scenarios when Stripe is under load and this lag time becomes on the order of minutes.Outstanding
This is likely also an issue for the "creation" of other streams' records, but since they are mutable, the record gets captured due to the
events_updates
pattern and causes them to be caught if missed.Keeping this scoped to only the immutable streams to keep the surface of the change low.
Manual QA steps
Risks
Rollback steps