[ADAP-1069] Custom batch_id prevents rebuilding the model on subsequent builds

[dlubawy]

Is this a new bug in dbt-bigquery?

• I believe this is a new bug in dbt-bigquery
• I have searched the existing issues, and I could not find an existing issue for this bug

Current Behavior

Setting a custom batch_id in a Python model's build config prevents rebuilds. Cause is a failure to create new batches due to conflicting IDs.

Expected Behavior

Setting a custom batch_id still allows for future dbt building of Python models.

Steps To Reproduce

1. Install v1.7 of dbt-core and dbt-bgiquery
2. Create a Python model with a custom batch_id configuration: dbt.config(..., batch_id='foo')
3. Build the Python model
4. Try building the Python model again

Relevant log output

409 Already exists: Failed to create batch: Batch projects/xxx-xxx-xxx/locations/xx-xxxxx/batches/foo

Environment

- OS: Darwin (macOS) 21.6.0
- Python: 3.9.18
- dbt-core: 1.7.3
- dbt-bigquery: 1.7.2

Additional Context

No response

[dlubawy]

This was found after trying to bypass the ADAP-1067 issue by setting a custom batch_id.

[dlubawy]

Looks like I missed that this was also found in ADAP-1016.

[colin-rogers-dbt]

Thanks for raising this @dlubawy, it's worth noting that the fix we have pushed out likely does not address this specific case as setting a static batch_id will be expected to fail on rerun (that's how dataproc batch Id's are supposed to work). Ultimately where we should go is probably having the config allow you to specify a batch_id prefix and we add the invocation_id as a suffix so that each run has a distinct batch_id.

[dlubawy]

Thanks for raising this @dlubawy, it's worth noting that the fix we have pushed out likely does not address this specific case as setting a static batch_id will be expected to fail on rerun (that's how dataproc batch Id's are supposed to work). Ultimately where we should go is probably having the config allow you to specify a batch_id prefix and we add the invocation_id as a suffix so that each run has a distinct batch_id.

Prefix with an appended invocation_id sounds like a good idea. However, there would need to be some enforcement on the user input for the configuration such that this wouldn't break due to batch_id length constraints. This is the issue for #1050 when batch_id was being derived from node names in v1.7.

[nathaniel-may]

This issue implies we have some opportunities to improve the UX around batch id.

[tanghyd]

I've been running into this problem for a while now. For completeness on issue tracking, I'll link my recent comment on a related issue: #1006.