Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

fix(ingestion): fix stateful ingestion for GCS source #11879

Merged
merged 1 commit into from
Jan 20, 2025
Merged

fix(ingestion): fix stateful ingestion for GCS source #11879

merged 1 commit into from
Jan 20, 2025

Conversation

josges
Copy link
Contributor

@josges josges commented Nov 18, 2024 •
edited
Loading

Remove pipeline name before passing context to equivalent s3 source to avoid error "Checkpointing provider DatahubIngestionCheckpointingProvider already registered." Fixes this issue

Checklist

  • The PR conforms to DataHub's Contributing Guideline (particularly Commit Message Format)
  • Links to related issues (if applicable)
  • Tests for the changes have been added/updated (if applicable)
  • Docs related to the changes have been added/updated (if applicable). If a new feature has been added a Usage Guide has been added for the same.
  • For any breaking change/potential downtime/deprecation/big changes an entry has been made in Updating DataHub

@github-actions github-actions bot added ingestion PR or Issue related to the ingestion of metadata community-contribution PR or Issue raised by member(s) of DataHub Community labels Nov 18, 2024
@hsheth2 hsheth2 requested a review from treff7es November 19, 2024 01:03
@datahub-cyborg datahub-cyborg bot added the needs-review Label for PRs that need review from a maintainer. label Nov 20, 2024
@datahub-cyborg datahub-cyborg bot added pending-submitter-response Issue/request has been reviewed but requires a response from the submitter and removed needs-review Label for PRs that need review from a maintainer. labels Dec 24, 2024
@josges josges force-pushed the fix_gcs branch 2 times, most recently from 931efb5 to c030f0e Compare December 27, 2024 12:20
@hsheth2 hsheth2 added needs-review Label for PRs that need review from a maintainer. and removed pending-submitter-response Issue/request has been reviewed but requires a response from the submitter labels Dec 31, 2024
@datahub-cyborg datahub-cyborg bot added pending-submitter-response Issue/request has been reviewed but requires a response from the submitter and removed needs-review Label for PRs that need review from a maintainer. labels Jan 3, 2025
@codecov Codecov
Copy link

codecov bot commented Jan 13, 2025 •
edited
Loading

Codecov Report

All modified and coverable lines are covered by tests ✅

Files with missing lines Coverage Δ
...ion/src/datahub/ingestion/source/gcs/gcs_source.py 93.50% <100.00%> (+0.08%) ⬆️

... and 1 file with indirect coverage changes

Continue to review full report in Codecov by Sentry.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 8d48622...1c6f040. Read the comment docs.

@datahub-cyborg datahub-cyborg bot added merge-pending-ci A PR that has passed review and should be merged once CI is green. and removed pending-submitter-response Issue/request has been reviewed but requires a response from the submitter labels Jan 13, 2025
@josges
fix(ingestion/gcs): fix stateful ingestion for GCS source
1c6f040 
Remove pipeline name before passing context to equivalent s3 source to avoid error "Checkpointing provider DatahubIngestionCheckpointingProvider already registered."
@mayurinehate mayurinehate merged commit 7ac6523 into datahub-project:master Jan 20, 2025
192 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@sgomezvillamor sgomezvillamor sgomezvillamor approved these changes

@mayurinehate mayurinehate mayurinehate approved these changes

@treff7es treff7es Awaiting requested review from treff7es

Assignees

@treff7es treff7es

Labels
community-contribution PR or Issue raised by member(s) of DataHub Community ingestion PR or Issue related to the ingestion of metadata merge-pending-ci A PR that has passed review and should be merged once CI is green.
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.

Ingestion for GCS ingest fails with stateful ingestion
5 participants
@josges @sgomezvillamor @mayurinehate @treff7es @hsheth2