-
Notifications
You must be signed in to change notification settings - Fork 4.3k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Iterable adjustable stream slices #8091
Conversation
29d673a
to
9c2021d
Compare
/test connector=connectors/source-iterable
|
/publish connector=connectors/source-iterable
|
f1e85f1
to
f01a49c
Compare
Resolves #7558
Some users experience problems with Iterable export stream could not complete cause thay have large data and API close connection before request finished.
To solve this issue introduce adjustable slice algorithm.
This algorithm is following:
previous request processing and request rate limit.
speed. Dividing this speed by 4 (request per minute limit) we can calculate
next slice range. Next range cannot be greater than 180 days
If processing of previous slice havent been completed, reset next range start date to previous slice
and reduce next slice range by 2 times.
In case if range havent been adjusted before getting next slice (it could
happend if there were no records for given date range), next slice would
have 180 days length.