[Bug]: PR 17365 breaks BQ loads in some cases

[steveniemitz]

What happened?

In beam 2.39, this used to work:

BigQueryIO
   .write()
   .optimizedWrites()
   .to(OutputTable)
   .withAvroFormatFunction(abc)
   .withAvroSchemaFactory(xyz)
   .withCreateDisposition(CreateDisposition.CREATE_NEVER)
   .withWriteDisposition(WriteDisposition.WRITE_TRUNCATE))

but now throws an exception if you end up in MultiPartitionsWriteTables:

Caused by: java.lang.IllegalArgumentException: Unless create disposition is CREATE_NEVER, a schema must be specified, i.e. DynamicDestinations.getSchema() may not return null. However, create disposition is CREATE_IF_NEEDED, and org.apache.beam.sdk.io.gcp.bigquery.DynamicDestinationsHelpers$ConstantTableDestinations@52e95e1 returned null for destination tableSpec: <table>
	at org.apache.beam.sdk.util.Preconditions.checkArgumentNotNull(Preconditions.java:436)
	at org.apache.beam.sdk.io.gcp.bigquery.WriteTables$WriteTablesDoFn.processElement(WriteTables.java:207)

The reason seems like now, the create disposition is set to CreateDisposition.CREATE_IF_NEEDED in WriteTables no matter what the user sets it to.

cc @pabloem @MarcoRob

Issue Priority

Priority: 2

Issue Component

Component: io-java-gcp

[steveniemitz]

Additionally, the new UpdateSchemaDestination breaks if the source format is set to avro due to it trying to load an empty file (a 0 length file is not a valid avro file).

[pabloem]

@johnjcasey @chamikaramj could you help look into this? I will be on vacation for a couple weeks from today.

[johnjcasey]

ack

[johnjcasey]

Agreed, this looks like that original PR changed how we load dynamic destinations for MultiPartitionsWrite tables. @MarcoRob can you take a look at this?

[MarcoRob]

Agreed, this looks like that original PR changed how we load dynamic destinations for MultiPartitionsWrite tables. @MarcoRob can you take a look at this?

Sure, let me check, tks

[Abacn]

Hi, just curious about what is the current status of this bug? Is the working example fixed by #22390 ? @steveniemitz @MarcoRob

[ahmedabu98]

Looks like #22390 fixed the Avro source problem referred to in the second comment.

[ahmedabu98]

Looks like the problem is in this deleted code block. The create disposition in WriteTables has always been CreateDisposition.CREATE_IF_NEEDED, because that's necessary when creating temp tables. However, in the case the schema isn't provided, this would cause the problem we're seeing here.

That code block used to handle this case by fetching the schema of the final destination table and using it as the schema for the temp tables. I think bringing that code back would solve this issue, unless there's a particular reason it was deleted @MarcoRob?

[MarcoRob]

Looks like the problem is in this deleted code block. The create disposition in WriteTables has always been CreateDisposition.CREATE_IF_NEEDED, because that's necessary when creating temp tables. However, in the case the schema isn't provided, this would cause the problem we're seeing here.

That code block used to handle this case by fetching the schema of the final destination table and using it as the schema for the temp tables. I think bringing that code back would solve this issue, unless there's a particular reason it was deleted @MarcoRob?

The change was part of a fix made due the following ticket.
I am taking a look at this now to validate the issue and fix the bug. I will be updating the bug with more info.

[Abacn]

Hi @MarcoRob, what is the status of this issue? Is it safe to restore the deleted block?

[Abacn]

Decide to reopen this issue because there is still more than one breakage remain

#25355

• incompatible schema error in copy when dynamicDestination
• check equality between provided schema and final destination schema does not work (#issuecomment-1423190926)

some fixes already done:
#22624 (fixes #22543) #24471 #24700

[Abacn]

#25355 is fixed and will be made available in upcoming 2.46.0. Close this for now.