Unable to respond with SQLAlchemy driver / Superset code base

[niravpeak]

Driver version

redshift-connector==2.0.907

Redshift version

Any => 1.0.38551 or redshift spectrum

Client Operating System

linux

Python version

Python 3.8.12

Table schema

NA

Problem description

1. Expected behaviour: Response should provide resultset
2. Actual behaviour: It throws error: First element of field tuple is neither a tuple nor str
3. Error message/stack trace:
   raceback (most recent call last):
   File "/app/superset/views/base.py", line 207, in wraps
   return f(self, *args, **kwargs)
   File "/app/superset/utils/log.py", line 245, in wrapper
   value = f(*args, **kwargs)
   File "/app/superset/views/core.py", line 2393, in sql_json
   command_result: CommandResult = command.run()
   File "/app/superset/sqllab/command.py", line 104, in run
   raise ex
   File "/app/superset/sqllab/command.py", line 96, in run
   status = self._run_sql_json_exec_from_scratch()
   File "/app/superset/sqllab/command.py", line 138, in _run_sql_json_exec_from_scratch
   raise ex
   File "/app/superset/sqllab/command.py", line 133, in _run_sql_json_exec_from_scratch
   return self._sql_json_executor.execute(
   File "/app/superset/sqllab/sql_json_executer.py", line 111, in execute
   raise SupersetErrorsException(
   superset.exceptions.SupersetErrorsException: [SupersetError(message='First element of field tuple is neither a tuple nor str', error_type=<SupersetErrorType.GENERIC_DB_ENGINE_ERROR: 'GENERIC_DB_ENGINE_ERROR'>, level=<ErrorLevel.ERROR: 'error'>, extra={'engine_name': 'Amazon Redshift', 'issue_codes': [{'code': 1002, 'message': 'Issue 1002 - The database returned an unexpected error.'}]})]
4. Any other details that can be helpful:
   Using IAM Based Authentication(Connect works perfect)| experimented with username password too it's failing too:
   URL: redshift+redshift_connector:///<database_name>
   conn_params = {
   "iam": true, # must be enabled when authenticating via IAM
   "cluster_identifier": "",
   "db_user": ''
   }

Python Driver trace logs

2022-05-17 05:50:12,363:DEBUG:redshift_connector:===================================
2022-05-17 05:50:12,363:DEBUG:redshift_connector.cursor:Cursor.paramstyle=named
2022-05-17 05:50:12,363:DEBUG:redshift_connector.core:===================================
2022-05-17 05:50:12,363:DEBUG:redshift_connector.core:Establishing a connection
2022-05-17 05:50:12,363:DEBUG:redshift_connector.core:{'user': 'IAM:uksegmentexplorer', 'database': 'dev', 'application_name': 'sqlalchemy-redshift', 'replication': None, 'client_protocol_version': '2', 'driver_version': 'Redshift Python Driver 2.0.907', 'os_version': 'Linux-5.4.181-99.354.amzn2.x86_64-x86_64-with-glibc2.2.5'}
2022-05-17 05:50:12,363:DEBUG:redshift_connector.core:===================================
2022-05-17 05:50:12,369:DEBUG:redshift_connector.cursor:Cursor.paramstyle=format
2022-05-17 05:50:12,369:DEBUG:redshift_connector.core:Sending start-up message
2022-05-17 05:50:12,551:DEBUG:redshift_connector.core:Server indicated EXTENDED_RESULT_METADATA transfer protocol will be used rather than protocol requested by client: BINARY
2022-05-17 05:50:12,551:DEBUG:redshift_connector.cursor:Cursor.paramstyle=format
2022-05-17 05:50:12,559:DEBUG:redshift_connector.core:field count=1

Reproduction code

Superset SQLAlchemy code: https://github.com/apache/superset

[Brooke-white]

Hey @niravpeak , thank you for opening this issue. Could you please provide a reproducible simple code snippit that shows the issue? We have not tested redshift_connector integration with apache superset, so this would be very helpful in my investigation as I haven't worked with superset before.

Please correct me if I am mistaken; My understanding of this issue is that it is regarding retrieving results from redshift_connector -- and that the results are not be in the form superset expects.

Another thing we can check here is if the statement being executed can execute using redshift_connector directly.

Thank you! :)

[Brooke-white]

Hi @niravpeak , following up on this issue. Could you please provide a reproducible simple code snippit so I can start to look into this?

[niravpeak]

@Brooke-white, It would be better if we have column name as same as other drivers. Reference apache/superset#20137

[Brooke-white]

Hi @niravpeak , thank you for this feedback. The superset folks have a PR opened to add support for redshift_connector, but a change to the column names from bytes to string would need to happen on the redshift_connector side. While this shouldn't block superset support, I will chat with the Redshift driver team about this request.

[Brooke-white]

Hi @niravpeak ,small update, we have a fix for this issue and are running it through our review process now. Assuming all is good, we expect to include this in our next release

[niravpeak]

I would be happy to assist on testing same if released/ available in this repo.

[Brooke-white]

Hey @niravpeak , we anticipate making a release including this change in early July. I will post an update here when that happens.

I saw the superset PR, 20137, was merged. Were you able to test that out?

[niravpeak]

Yes @Brooke-white it has worked 👍

[Brooke-white]

Hi @niravpeak -- we've released v2.0.908 today which supports receiving column names as str. As such, I'll be closing this issue, but feel free to open a new one if you see any issue or need additional help. Thanks again for bringing this to our attention so we could work to resolve it :)