Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Iterable adjustable stream slices #8091

Merged
merged 3 commits into from
Nov 22, 2021
Merged

Conversation

avida
Copy link
Contributor

@avida avida commented Nov 18, 2021

Resolves #7558

Some users experience problems with Iterable export stream could not complete cause thay have large data and API close connection before request finished.

To solve this issue introduce adjustable slice algorithm.

This algorithm is following:

  1. First slice have 30 days length.
  2. When slice is processed by stream next slice range is calculated based on how much time took
    previous request processing and request rate limit.
  3. Knowing previous slice range we can calculate days per minute processing
    speed. Dividing this speed by 4 (request per minute limit) we can calculate
    next slice range. Next range cannot be greater than 180 days

If processing of previous slice havent been completed, reset next range start date to previous slice
and reduce next slice range by 2 times.

In case if range havent been adjusted before getting next slice (it could
happend if there were no records for given date range), next slice would
have 180 days length.

@github-actions github-actions bot added the area/connectors Connector related issues label Nov 18, 2021
@avida avida changed the title iterable fix Draft: Iterable adjustable stream slices Nov 18, 2021
@avida avida temporarily deployed to more-secrets November 18, 2021 12:43 Inactive
@avida avida force-pushed the drezchykov/iterable-adj-slices branch from 29d673a to 9c2021d Compare November 19, 2021 15:10
@avida avida temporarily deployed to more-secrets November 19, 2021 15:12 Inactive
@avida avida marked this pull request as ready for review November 19, 2021 15:17
@avida avida changed the title Draft: Iterable adjustable stream slices Iterable adjustable stream slices Nov 19, 2021
@avida
Copy link
Contributor Author

avida commented Nov 19, 2021

/test connector=connectors/source-iterable

🕑 connectors/source-iterable https://github.com/airbytehq/airbyte/actions/runs/1481612632
✅ connectors/source-iterable https://github.com/airbytehq/airbyte/actions/runs/1481612632
Python tests coverage:

	 ---------- coverage: platform linux, python 3.8.10-final-0 -----------
	 Name                                                 Stmts   Miss  Cover
	 ------------------------------------------------------------------------
	 source_acceptance_test/__init__.py                       2      0   100%
	 source_acceptance_test/base.py                          10      4    60%
	 source_acceptance_test/config.py                        75      8    89%
	 source_acceptance_test/conftest.py                     108    108     0%
	 source_acceptance_test/plugin.py                        47     47     0%
	 source_acceptance_test/tests/__init__.py                 4      0   100%
	 source_acceptance_test/tests/test_core.py              200     94    53%
	 source_acceptance_test/tests/test_full_refresh.py       38     27    29%
	 source_acceptance_test/tests/test_incremental.py        69     38    45%
	 source_acceptance_test/utils/__init__.py                 6      0   100%
	 source_acceptance_test/utils/asserts.py                 37      2    95%
	 source_acceptance_test/utils/common.py                  41     24    41%
	 source_acceptance_test/utils/compare.py                 62     25    60%
	 source_acceptance_test/utils/connector_runner.py        82     49    40%
	 source_acceptance_test/utils/json_schema_helper.py     115     14    88%
	 ------------------------------------------------------------------------
	 TOTAL                                                  896    440    51%
	 ---------- coverage: platform linux, python 3.8.10-final-0 -----------
	 Name                                  Stmts   Miss  Cover
	 ---------------------------------------------------------
	 source_iterable/__init__.py               2      0   100%
	 source_iterable/api.py                  159     69    57%
	 source_iterable/iterable_streams.py     107     15    86%
	 source_iterable/slice_generators.py      73      0   100%
	 source_iterable/source.py                15      6    60%
	 ---------------------------------------------------------
	 TOTAL                                   356     90    75%

@jrhizor jrhizor temporarily deployed to more-secrets November 19, 2021 15:27 Inactive
@github-actions github-actions bot added the area/documentation Improvements or additions to documentation label Nov 22, 2021
@avida
Copy link
Contributor Author

avida commented Nov 22, 2021

/publish connector=connectors/source-iterable

🕑 connectors/source-iterable https://github.com/airbytehq/airbyte/actions/runs/1490328082
✅ connectors/source-iterable https://github.com/airbytehq/airbyte/actions/runs/1490328082

@avida avida temporarily deployed to more-secrets November 22, 2021 13:18 Inactive
@jrhizor jrhizor temporarily deployed to more-secrets November 22, 2021 13:19 Inactive
@avida avida force-pushed the drezchykov/iterable-adj-slices branch from f1e85f1 to f01a49c Compare November 22, 2021 13:47
@avida avida merged commit cf3be4a into master Nov 22, 2021
@avida avida deleted the drezchykov/iterable-adj-slices branch November 22, 2021 13:48
@avida avida temporarily deployed to more-secrets November 22, 2021 13:49 Inactive
schlattk pushed a commit to schlattk/airbyte that referenced this pull request Jan 4, 2022
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
area/connectors Connector related issues area/documentation Improvements or additions to documentation
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Iterable Sync timeout/cancelation due to Memory Error/Heartbeat Failure
3 participants