Destination Redshift fails after upgrade

[m-ronchi]

Environment

• Airbyte version: example is 0.38.2-alpha
• OS Version / Instance:
• Deployment: Kubernetes
• Source Connector and version: any
• Destination Connector and version: Redshift 0.3.33
• Severity: Medium
• Step where error happened: Sync job

Current Behavior

I upgraded airbyte from 0.35 to 0.38, which also upgraded connectors. Redshift destination went from 0.3.23 to 0.3.32 (I also manually tried 0.3.33 but the problem persisted)

I found out that now redshift destination forces all _airbyte_raw_* tables to use the SUPER type instead of VARCHAR, but that causes the destination to fail on close with the exceptions below.

I also noticed that the relevant PRs say that is an option, but I can't find anything to disable it and return to the old behaviour.

Expected Behavior

syncs to Redshift still work after upgrade

Logs

If applicable, please upload the logs from the failing operation.
For sync jobs, you can download the full logs from the UI by going to the sync attempt page and
clicking the download logs button at the top right of the logs display window.

LOG

logs-6007.txt
logs-6008.txt

Steps to Reproduce

1. sync with old redshift versions. I have multiple connections in the same schema (with different prefixes)
2. upgrade redshift
3. sync again

[marcosmarxm]

Can you check this issue @m-ronchi #12265

[alexandertsukanov]

@m-ronchi any updates on the issue?
CC: @marcosmarxm

[m-ronchi]

we are keeping redshift at 0.3.29.

it may be that the error is caused on custom views that we have created on some of the _airbyte_raw tables.

the problem is that the conversion is forced on every table in the DB whose name matches your pattern, including unrelated ones.

[alexandertsukanov]

@m-ronchi could you to provide the example (name) of effected table(s), please?

[m-ronchi]

you can see them in the logs attached.

e.g. logs-6008.txt is from a google ads sync, but it fails on a facebook marketing table (and it attempts to convert all other tables in the database)

you should either:

• make the SUPER columns (and the conversion) actually optional as it is stated in the docs
• only touch the specific tables for the streams you are syncing (then if something fails we can at least rename them and manually fix downstream)
• or both

[alexandertsukanov]

@m-ronchi the issue was automatically closed after PR had been merged. Please, verify if the fix the work for you, and re-open if it isn't. Much appreciated. Thanks.