Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat: Add support for new BATCH message type in taps and targets #904

Merged
merged 18 commits into from
Sep 15, 2022

Conversation

edgarrmondragon
Copy link
Collaborator

@edgarrmondragon edgarrmondragon commented Aug 10, 2022

@edgarrmondragon edgarrmondragon changed the title Add built-in support for BATCH message type (aka FAST_SYNC spec) feat: Add built-in support for BATCH message type (aka FAST_SYNC spec) Aug 10, 2022
@edgarrmondragon edgarrmondragon force-pushed the feat/batch-messages branch 6 times, most recently from d685a8c to 4d28791 Compare August 11, 2022 15:00
@meltano meltano deleted a comment from codecov bot Aug 11, 2022
@codecov
Copy link

codecov bot commented Aug 11, 2022

Codecov Report

Merging #904 (42d980b) into main (aab68a1) will increase coverage by 0.59%.
The diff coverage is 90.90%.

@@            Coverage Diff             @@
##             main     #904      +/-   ##
==========================================
+ Coverage   81.18%   81.77%   +0.59%     
==========================================
  Files          36       37       +1     
  Lines        3555     3742     +187     
  Branches      718      746      +28     
==========================================
+ Hits         2886     3060     +174     
- Misses        492      500       +8     
- Partials      177      182       +5     
Impacted Files Coverage Δ
singer_sdk/io_base.py 76.47% <66.66%> (-1.31%) ⬇️
singer_sdk/mapper_base.py 56.14% <66.66%> (+0.58%) ⬆️
singer_sdk/sinks/sql.py 78.48% <80.00%> (-0.47%) ⬇️
singer_sdk/helpers/_batch.py 90.38% <90.38%> (ø)
singer_sdk/sinks/core.py 80.13% <90.90%> (+1.39%) ⬆️
singer_sdk/streams/core.py 83.25% <92.06%> (+1.87%) ⬆️
singer_sdk/helpers/_singer.py 96.59% <100.00%> (+0.17%) ⬆️
singer_sdk/target_base.py 85.37% <100.00%> (+0.35%) ⬆️
singer_sdk/streams/rest.py 83.87% <0.00%> (+1.29%) ⬆️

📣 We’re building smart automated test selection to slash your CI/CD build times. Learn more

@edgarrmondragon edgarrmondragon changed the base branch from main to feat/BATCH August 11, 2022 23:21
@edgarrmondragon edgarrmondragon changed the base branch from feat/BATCH to main August 17, 2022 22:44
@edgarrmondragon edgarrmondragon force-pushed the feat/batch-messages branch 4 times, most recently from 0e4bc33 to 6752917 Compare August 23, 2022 03:53
@edgarrmondragon edgarrmondragon force-pushed the feat/batch-messages branch 8 times, most recently from 080eefe to b88f56a Compare August 30, 2022 23:07
@edgarrmondragon edgarrmondragon marked this pull request as ready for review August 30, 2022 23:15
@edgarrmondragon edgarrmondragon requested a review from a team as a code owner September 13, 2022 22:56
@edgarrmondragon
Copy link
Collaborator Author

@kgpayne @aaronsteers this is ready for final (🤞) review 😄

docs/batch.md Outdated Show resolved Hide resolved
@aaronsteers aaronsteers changed the title feat: Add built-in support for BATCH message type (aka FAST_SYNC spec) feat: Add new BATCH message type support for taps and targets Sep 13, 2022
@aaronsteers aaronsteers changed the title feat: Add new BATCH message type support for taps and targets feat: Add support for new BATCH message type in taps and targets Sep 13, 2022
Copy link
Contributor

@aaronsteers aaronsteers left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@edgarrmondragon - I made two small tweaks in the docs page. For my part, I think this is ready to go. Fantastic work! 🚀

@aaronsteers aaronsteers modified the milestone: v1.0 Release Sep 14, 2022
docs/batch.md Outdated Show resolved Hide resolved
Co-authored-by: Aaron ("AJ") Steers <[email protected]>
@edgarrmondragon
Copy link
Collaborator Author

@aaronsteers target now extracts storage root from absolute URLs presented in the manifest array: bf06a9c

  • Moved all the batch helpers to singer_sdk.helpers._batch to reduce clutter in singer_sdk.helpers._singer
  • Storage object can now be built from a file URL

Copy link
Contributor

@aaronsteers aaronsteers left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@edgarrmondragon - A few docs tweaks proposed. Nothing here is blocking but hopefully these small tweaks increase clarity overall.

docs/batch.md Outdated Show resolved Hide resolved
docs/batch.md Outdated Show resolved Hide resolved
docs/batch.md Outdated Show resolved Hide resolved
docs/batch.md Outdated Show resolved Hide resolved
@aaronsteers
Copy link
Contributor

@edgarrmondragon - Thanks very much for the work to pass and accept absolute paths. I proposed a few docs updates. Feel free to merge when ready, per mine and @cjohnhanson's approval here on the PR.

Co-authored-by: Aaron ("AJ") Steers <[email protected]>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

feature: Add built-in support for BATCH message type (aka FAST_SYNC spec)
3 participants