Add DLQ support to JdbcToBigQuery using BQ Storage Write API #2016
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Codecov ReportAttention: Patch coverage is
Additional details and impacted files@@ Coverage Diff @@
## main #2016 +/- ##
============================================
+ Coverage 45.51% 45.56% +0.05%
- Complexity 3704 4047 +343
============================================
Files 845 847 +2
Lines 50084 50213 +129
Branches 5268 5279 +11
============================================
+ Hits 22794 22879 +85
- Misses 25618 25654 +36
- Partials 1672 1680 +8
|
|
Re-triggered failed workflows (Java PR in particular did not run full IT suite) |
damccorm
previously approved these changes
Nov 20, 2024
Polber
force-pushed
the
jkinard/jdbc-to-bigquery-dlq
branch
from
109905d
to
6a04393
Compare
Abacn
reviewed
Nov 21, 2024
v2/jdbc-to-googlecloud/src/main/java/com/google/cloud/teleport/v2/templates/JdbcToBigQuery.java
Show resolved
Hide resolved
Abacn
previously approved these changes
Nov 22, 2024
Signed-off-by: Jeffrey Kinard <[email protected]>
Signed-off-by: Jeffrey Kinard <[email protected]>
Polber
force-pushed
the
jkinard/jdbc-to-bigquery-dlq
branch
from
6a04393
to
45ae79e
Compare
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
This PR adds proper error handling to the JdbcToBigQueryFlex template when using Storage Write API.
Before this change, the WriteToBigQuery step would succeed even when there were write failures. These errors were never captured or handled, thus resulting in dropped rows. This PR will now fail the pipeline on write errors by default when using Storage Write API. If a deadletter table is specified, then errors will instead be written the the DLQ table (created if needed).