[FLIGHT][GO] CommandGetTables metadata being ignored

[xborder]

What would you like help with?

Hi,

I'm testing a connector that uses ADBC Go driver against a flight server. While I can obtain information about catalogs, schemas and tables, when it comes to the columns everything is empty.

I believe the issue is with the GO driver and compliance with FlightSQL but I'd like to get some validation on my interpretation.
FlightSQL specification states the following for CommandGetTables response:

 ....

  The returned Arrow schema will be:
  <
   catalog_name: utf8,
   db_schema_name: utf8,
   table_name: utf8 not null,
   table_type: utf8 not null,
   [optional] table_schema: bytes not null (schema of the table as described in Schema.fbs::Schema,
                                            it is serialized as an IPC message.)
  >
  Fields on table_schema may contain the following metadata:
   - ARROW:FLIGHT:SQL:CATALOG_NAME      - Table's catalog name
   - ARROW:FLIGHT:SQL:DB_SCHEMA_NAME    - Database schema name
   - ARROW:FLIGHT:SQL:TABLE_NAME        - Table name
   - ARROW:FLIGHT:SQL:TYPE_NAME         - The data source-specific name for the data type of the column.
   - ARROW:FLIGHT:SQL:PRECISION         - Column precision/size
   - ARROW:FLIGHT:SQL:SCALE             - Column scale/decimal digits if applicable
   - ARROW:FLIGHT:SQL:IS_AUTO_INCREMENT - "1" indicates if the column is auto incremented, "0" otherwise.
   - ARROW:FLIGHT:SQL:IS_CASE_SENSITIVE - "1" indicates if the column is case-sensitive, "0" otherwise.
   - ARROW:FLIGHT:SQL:IS_READ_ONLY      - "1" indicates if the column is read only, "0" otherwise.
   - ARROW:FLIGHT:SQL:IS_SEARCHABLE     - "1" indicates if the column is searchable via WHERE clause, "0" otherwise.
....

• I can see this information being returned to the driver so the server I'm testing against is respecting the specification.
• Regardless of the server returns, the result of GetObjects doesn't have type information about the existing columns
• While debugging the driver I think the issue is in how the column information is being processed (in [shared_utils.go](https://github.com/apache/arrow-adbc/blob/c3149c0746d09877709e7625b7199e102d6d9c94/go/adbc/driver/internal/shared_utils.go#L378))
• If there's the columns contain metadata, there is no code looking for the metadata keys described in the specification
• JAVA ADBC seems to have a different implementation where these metadata keys are used

Questions:

1. Does this mean that GO driver currently is not respecting the FlightSQL spec?
2. If this is the case what should it look for in the metadata? The ARROW:FLIGHT:SQL: keys described in the spec or the xdbc_ keys? Should it look for ARROW:FLIGHT:SQL: and fallback to xdbc_?
3. Where can I get more information about these xdbc_ metadata fields?

Thank you

[zeroshade]

Looks like you're absolutely correct, we're not properly propagating the metadata from flight sql.

If this is the case what should it look for in the metadata? The ARROW:FLIGHT:SQL: keys described in the spec or the xdbc_ keys? Should it look for ARROW:FLIGHT:SQL: and fallback to xdbc_?

It should definitely be using the correct ARROW:FLIGHT:SQL:* keys and fallback to looking for the XDBC_* ones. I'll look into fixing this when I have the opportunity. That said, if you would like to fix it yourself, i'll happily review it!

[xborder]

Cool, thank you for the confirmation. I'll try to fix it

[davidhcoe]

I think the genesis of this was as a utility function for non-FlightSQL databases.