Troubleshooting Guide #176

prashastia · 2024-10-27T15:57:46Z

/gcbrun

…sage.

...igquery/src/main/java/com/google/cloud/flink/bigquery/sink/writer/BigQueryDefaultWriter.java

prashastia · 2024-10-29T06:32:18Z

...ommon/src/main/java/com/google/cloud/flink/bigquery/common/utils/BigQueryPartitionUtils.java

@@ -264,7 +264,7 @@ static String timestampRestrictionFromPartitionType(
                    // extract a datetime from the value and restrict
                    // between previous and next hour


I know support for unbounded read is going to be removed. But this was a very simple and easy change to fix reading of Time Partitions based on DAY, MONTH and YEAR.

...va/com/google/cloud/flink/bigquery/source/reader/deserializer/AvroDeserializationSchema.java

...ogle/cloud/flink/bigquery/source/reader/deserializer/AvroToRowDataDeserializationSchema.java

...om/google/cloud/flink/bigquery/source/reader/deserializer/BigQueryDeserializationSchema.java

TROUBLESHOOT.md

jayehwhyehentee · 2024-10-29T06:45:08Z

TROUBLESHOOT.md

+- It is also expected that the value passed in the Avro Generic Record follows the Schema.  
+Here the “records passed” indicates the modified records after passing through the series of 
+subtasks defined in the application pipeline.


This info is well captured in preceding and next points

Value in the avro record might differ from the avro schema of the Generic Record (since Flink Does not impose any check on the value of the field) This is the error faced by GMF very early on in testing the connector when they accidentally passed the wrong value INTEGER in an ARRAY type field.

That is a problem where we must prompt the user to ensure that their destination table matches with the schema of records received by the sink.

The statement we should remove is ..

Here the “records passed” indicates the modified records after passing through the series of subtasks defined in the application pipeline.

.. because this doesn't prompt a schema check, instead says something fairly obvious that doesn't add much value

Fair, removed.

TROUBLESHOOT.md

…rite.

jayehwhyehentee · 2024-11-18T07:19:19Z

...va/com/google/cloud/flink/bigquery/source/reader/deserializer/AvroDeserializationSchema.java


    public AvroDeserializationSchema(String avroSchemaString) {
        this.avroSchemaString = avroSchemaString;
    }

    @Override
-    public GenericRecord deserialize(GenericRecord record) throws IOException {
+    public GenericRecord deserialize(GenericRecord record) throws BigQueryConnectorException {


You dont need to throw any exception in this method's definition

jayehwhyehentee · 2024-11-18T07:23:32Z

...om/google/cloud/flink/bigquery/source/reader/deserializer/BigQueryDeserializationSchema.java

+        try {
+            if (deserialize != null) {
+                out.collect(deserialize);
+            }
+        } catch (Exception e) {
+            LOG.error(
+                    String.format(
+                            "Failed to forward the deserialized record %s to the next operator.%nError %s%nCause %s",
+                            deserialize, e.getMessage(), e.getCause()));
+            throw new BigQueryConnectorException(
+                    "Failed to forward the deserialized record to the next operator.", e);


Reduce nesting as much as you can.
How about:

if (deserialize == null) { return; } try { out.collect(deserialize); } catch (Exception e) { ...

jayehwhyehentee · 2024-11-18T07:24:39Z

...gquery/src/main/java/com/google/cloud/flink/bigquery/sink/writer/BigQueryBufferedWriter.java

-        // Reset the "Since Checkpoint" values to 0.
-        numberOfRecordsBufferedByBigQuerySinceCheckpoint.dec(
-                numberOfRecordsBufferedByBigQuerySinceCheckpoint.getCount());
-        numberOfRecordsSeenByWriterSinceCheckpoint.dec(
-                numberOfRecordsSeenByWriterSinceCheckpoint.getCount());


This change is being tracked in a separate PR. Please remove from here

jayehwhyehentee · 2024-11-18T07:27:10Z

TROUBLESHOOT.md

+- The problem lies with the pipeline, the previous chain of subtasks that are performed before 
+sink is called.
+- The pipeline is not processing and passing the records forward for the sink.


Most likely not an issue in the sink, since previous subtasks are not passing records forward for the sink.

jayehwhyehentee · 2024-11-18T07:28:38Z

TROUBLESHOOT.md

+#### The records are arriving at the sink but not being successfully written to BigQuery.
+Check the logs or error message for the following errors:
+#### `BigQuerySerializationException`
+- This message illustrates that the record(s) could not be serialized by the connector. 


record not record(s) since serialize exception for every record will be logged individually

jayehwhyehentee · 2024-11-18T07:28:48Z

TROUBLESHOOT.md

+Check the logs or error message for the following errors:
+#### `BigQuerySerializationException`
+- This message illustrates that the record(s) could not be serialized by the connector. 
+- The error message would also contain the actual cause for the same.


jayehwhyehentee · 2024-11-18T07:30:19Z

TROUBLESHOOT.md

+- Note: This error is not thrown but logged, 
+indicating that the connector was "Unable to serialize record" due to this error.


- This error is logged not thrown, explaining why the record could not be serialized. - In future, this will be supplemented with dead letter queues.

Also, please mention logged not thrown in bold.

jayehwhyehentee · 2024-11-18T07:34:38Z

TROUBLESHOOT.md

+- It is also expected that the value passed in the Avro Generic Record follows the Schema.  
+Here the “records passed” indicates the modified records after passing through the series of 
+subtasks defined in the application pipeline.


That is a problem where we must prompt the user to ensure that their destination table matches with the schema of records received by the sink.

The statement we should remove is ..

Here the “records passed” indicates the modified records after passing through the series of subtasks defined in the application pipeline.

.. because this doesn't prompt a schema check, instead says something fairly obvious that doesn't add much value

prashastia · 2024-11-19T06:22:22Z

@clmccart Pls review this PR. Thanks!

jayehwhyehentee

LGTM. Please merge after @clmccart's approval

clmccart

are the changes that arent the troubleshooting guide supposed to be included in this PR?

clmccart · 2024-12-06T22:36:55Z

TROUBLESHOOT.md

+### Records are not being written to BigQuery
+With the help of metrics available as a part of 0.4.0 release of the connector, 
+users should be able to track the number of records that enter the sink(writer) and the 
+number of records successfully written to BigQuery. 


lets reference the two metric names here

clmccart · 2024-12-06T22:38:10Z

TROUBLESHOOT.md

+users should not face this error.
+Users might face this error in case custom serializer is used.
+
+## Known Issues/Limitations


let's also reference the 100 max parallelism here

We have mentioned this in the Readme, should we mention it here as well?

prashastia · 2024-12-08T17:25:17Z

are the changes that aren't the troubleshooting guide supposed to be included in this PR?

@clmccart Yep, some error messages documented in the troubleshooting guide and minor bugs are fixed as a part of these PR as well.

prashastia added 13 commits September 23, 2024 10:50

Populate User Agent Header and Trace ID to track Read and Write API u…

300194b

…sage.

Populate User Agent Header and Trace ID to track Read and Write API u…

c11c8ee

…sage.

Populate User Agent Header and Trace ID to track Read and Write API u…

4e75e1e

…sage.

Remove the commented out .setTraceId() for

f30c973

Remove User Agent Header from the transport channel.

8549563

throw BigQueryConnector Exception in case of schema mismatch.

7f08506

throw BigQueryConnector Exception in case of schema mismatch.

e4c3f49

Fix No edge case time type.

72d0db2

Merge remote-tracking branch 'dataproc/main' into improve-debuggability

a652677

Remove log messages.

df4741f

Fix day, month and year partition reads along with tests.

64c38c8

Merge remote-tracking branch 'dataproc/main' into troubleshooting-guide

cefd02f

Add TROUBLESHOOT.md

80e8470

jayehwhyehentee reviewed Oct 29, 2024

View reviewed changes

...igquery/src/main/java/com/google/cloud/flink/bigquery/sink/writer/BigQueryDefaultWriter.java Outdated Show resolved Hide resolved

prashastia commented Oct 29, 2024

View reviewed changes

jayehwhyehentee reviewed Oct 29, 2024

View reviewed changes

prashastia added 2 commits October 29, 2024 12:30

Merge remote-tracking branch 'dataproc/main' into troubleshooting-guide

99903c6

Address review changes for code.

e7fd9ea

prashastia mentioned this pull request Oct 29, 2024

Improve debugability #164

Closed

prashastia added 5 commits October 29, 2024 13:28

Address review changes for code.

5fddee2

Merge remote-tracking branch 'dataproc/main' into troubleshooting-guide

39c2ea0

Address review changes for code.

250240c

Remove extra space.

2d179c4

Address Review Comments - elaborate BigQueryConnector Exception.

6acf4d1

prashastia requested a review from jayehwhyehentee November 7, 2024 04:12

prashastia added 2 commits November 12, 2024 15:59

Reset Since Checkpoint variables in SnapshotState rather than first w…

5aa3b10

…rite.

Merge remote-tracking branch 'dataproc/main' into troubleshooting-guide

89050de

prashastia force-pushed the troubleshooting-guide branch from 89b1e1a to 89050de Compare November 12, 2024 11:35

jayehwhyehentee reviewed Nov 18, 2024

View reviewed changes

Address review comments.

c126f5a

Address review comments.

4e52d6a

jayehwhyehentee self-requested a review November 19, 2024 11:49

jayehwhyehentee approved these changes Nov 19, 2024

View reviewed changes

clmccart reviewed Dec 6, 2024

View reviewed changes

Address review comments.

dc2babf

prashastia requested a review from clmccart December 10, 2024 05:29

jayehwhyehentee merged commit 8d519aa into GoogleCloudDataproc:main Dec 11, 2024
4 checks passed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Troubleshooting Guide #176

Troubleshooting Guide #176

prashastia commented Oct 27, 2024

prashastia Oct 29, 2024

jayehwhyehentee Oct 29, 2024

jayehwhyehentee Oct 29, 2024

prashastia Nov 6, 2024

jayehwhyehentee Nov 18, 2024

prashastia Nov 19, 2024

jayehwhyehentee Nov 18, 2024

prashastia Nov 19, 2024

jayehwhyehentee Nov 18, 2024

prashastia Nov 19, 2024

jayehwhyehentee Nov 18, 2024

prashastia Nov 19, 2024

jayehwhyehentee Nov 18, 2024

prashastia Nov 19, 2024

jayehwhyehentee Nov 18, 2024

prashastia Nov 19, 2024

jayehwhyehentee Nov 18, 2024

jayehwhyehentee Nov 18, 2024

prashastia Nov 19, 2024

jayehwhyehentee Nov 18, 2024

prashastia commented Nov 19, 2024

jayehwhyehentee left a comment

clmccart left a comment

clmccart Dec 6, 2024

prashastia Dec 8, 2024

clmccart Dec 6, 2024

prashastia Dec 8, 2024

prashastia commented Dec 8, 2024 •

edited

Loading

		@@ -264,7 +264,7 @@ static String timestampRestrictionFromPartitionType(
		// extract a datetime from the value and restrict
		// between previous and next hour

		- Note: This error is not thrown but logged,
		indicating that the connector was "Unable to serialize record" due to this error.

Troubleshooting Guide #176

Troubleshooting Guide #176

Conversation

prashastia commented Oct 27, 2024

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

prashastia commented Nov 19, 2024

jayehwhyehentee left a comment

Choose a reason for hiding this comment

clmccart left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

prashastia commented Dec 8, 2024 • edited Loading

prashastia commented Dec 8, 2024 •

edited

Loading