python/postgres: no binary output function available for type aclitem

[bigluck]

What happened?

With the latest version of the library (1.4.0) this query:

SELECT n.oid,n.*,d.description FROM pg_catalog.pg_namespace n LEFT OUTER JOIN pg_catalog.pg_description d ON d.objoid=n.oid AND d.objsubid=0 AND d.classoid='pg_namespace'::regclass ORDER BY nspname

fails with:

adbc_driver_manager.ProgrammingError: INVALID_ARGUMENT: [libpq] Failed to execute query 'SELECT n.oid,n.*,d.description FROM pg_catalog.pg_namespace n LEFT OUTER JOIN pg_catalog.pg_description d ON d.objoid=n.oid AND d.objsubid=0 AND d.classoid='pg_namespace'::regclass ORDER BY nspname': ERROR:  no binary output function available for type aclitem

I can't control the input SQL query, but I could "disable", if it's possible, the binary mode for all the queries against the pg_catalog namespace, but it isn't supported.

Stack Trace

[postgres   ] 2025-03-03 15:06:30.901 GMT [56969] ERROR:  no binary output function available for type aclitem
[postgres   ] 2025-03-03 15:06:30.901 GMT [56969] STATEMENT:  SELECT n.oid,n.*,d.description FROM pg_catalog.pg_namespace n LEFT OUTER JOIN pg_catalog.pg_description d ON d.objoid=n.oid AND d.objsubid=0 AND d.classoid='pg_namespace'::regclass ORDER BY nspname
[postgres   ] 2025-03-03 15:06:30.907 GMT [56969] WARNING:  there is no transaction in progress

Traceback (most recent call last):
  File "/Users/bigluck/dev/.venv/lib/python3.12/site-packages/anyio/streams/memory.py", line 105, in receive
    return self.receive_nowait()
           ^^^^^^^^^^^^^^^^^^^^^
  File "/Users/bigluck/dev/.venv/lib/python3.12/site-packages/anyio/streams/memory.py", line 100, in receive_nowait
    raise WouldBlock
anyio.WouldBlock

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/Users/bigluck/dev/.venv/lib/python3.12/site-packages/anyio/streams/memory.py", line 118, in receive
    return receiver.item
           ^^^^^^^^^^^^^
AttributeError: 'MemoryObjectItemReceiver' object has no attribute 'item'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/Users/bigluck/dev/.venv/lib/python3.12/site-packages/starlette/middleware/base.py", line 157, in call_next
    message = await recv_stream.receive()
              ^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/bigluck/dev/.venv/lib/python3.12/site-packages/anyio/streams/memory.py", line 120, in receive
    raise EndOfStream
anyio.EndOfStream

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/Users/bigluck/dev/src/main.py", line 104, in catch_exceptions_middleware
    out = await call_next(request)
          ^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/bigluck/dev/.venv/lib/python3.12/site-packages/starlette/middleware/base.py", line 163, in call_next
    raise app_exc
  File "/Users/bigluck/dev/.venv/lib/python3.12/site-packages/starlette/middleware/base.py", line 149, in coro
    await self.app(scope, receive_or_disconnect, send_no_error)
  File "/Users/bigluck/dev/.venv/lib/python3.12/site-packages/starlette/middleware/base.py", line 185, in __call__
    with collapse_excgroups():
  File "/nix/store/icv5w5d2sl202gxpz98hl33hglkir5c0-python3-3.12.3/lib/python3.12/contextlib.py", line 158, in __exit__
    self.gen.throw(value)
  File "/Users/bigluck/dev/.venv/lib/python3.12/site-packages/starlette/_utils.py", line 82, in collapse_excgroups
    raise exc
  File "/Users/bigluck/dev/.venv/lib/python3.12/site-packages/starlette/middleware/base.py", line 187, in __call__
    response = await self.dispatch_func(request, call_next)
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/bigluck/dev/.venv/lib/python3.12/site-packages/slowapi/middleware.py", line 136, in dispatch
    response = await call_next(request)
               ^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/bigluck/dev/.venv/lib/python3.12/site-packages/starlette/middleware/base.py", line 163, in call_next
    raise app_exc
  File "/Users/bigluck/dev/.venv/lib/python3.12/site-packages/starlette/middleware/base.py", line 149, in coro
    await self.app(scope, receive_or_disconnect, send_no_error)
  File "/Users/bigluck/dev/.venv/lib/python3.12/site-packages/starlette/middleware/gzip.py", line 20, in __call__
    await responder(scope, receive, send)
  File "/Users/bigluck/dev/.venv/lib/python3.12/site-packages/starlette/middleware/gzip.py", line 39, in __call__
    await self.app(scope, receive, self.send_with_gzip)
  File "/Users/bigluck/dev/.venv/lib/python3.12/site-packages/opentelemetry/instrumentation/asgi/__init__.py", line 743, in __call__
    await self.app(scope, otel_receive, otel_send)
  File "/Users/bigluck/dev/.venv/lib/python3.12/site-packages/starlette/middleware/cors.py", line 85, in __call__
    await self.app(scope, receive, send)
  File "/Users/bigluck/dev/.venv/lib/python3.12/site-packages/starlette/middleware/exceptions.py", line 62, in __call__
    await wrap_app_handling_exceptions(self.app, conn)(scope, receive, send)
  File "/Users/bigluck/dev/.venv/lib/python3.12/site-packages/starlette/_exception_handler.py", line 53, in wrapped_app
    raise exc
  File "/Users/bigluck/dev/.venv/lib/python3.12/site-packages/starlette/_exception_handler.py", line 42, in wrapped_app
    await app(scope, receive, sender)
  File "/Users/bigluck/dev/.venv/lib/python3.12/site-packages/starlette/routing.py", line 715, in __call__
    await self.middleware_stack(scope, receive, send)
  File "/Users/bigluck/dev/.venv/lib/python3.12/site-packages/starlette/routing.py", line 735, in app
    await route.handle(scope, receive, send)
  File "/Users/bigluck/dev/.venv/lib/python3.12/site-packages/starlette/routing.py", line 288, in handle
    await self.app(scope, receive, send)
  File "/Users/bigluck/dev/.venv/lib/python3.12/site-packages/starlette/routing.py", line 76, in app
    await wrap_app_handling_exceptions(app, request)(scope, receive, send)
  File "/Users/bigluck/dev/.venv/lib/python3.12/site-packages/starlette/_exception_handler.py", line 53, in wrapped_app
    raise exc
  File "/Users/bigluck/dev/.venv/lib/python3.12/site-packages/starlette/_exception_handler.py", line 42, in wrapped_app
    await app(scope, receive, sender)
  File "/Users/bigluck/dev/.venv/lib/python3.12/site-packages/starlette/routing.py", line 73, in app
    response = await f(request)
               ^^^^^^^^^^^^^^^^
  File "/Users/bigluck/dev/src/router.py", line 86, in custom_route_handler
    out: Response = await original_route_handler(request)
                    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/bigluck/dev/.venv/lib/python3.12/site-packages/fastapi/routing.py", line 301, in app
    raw_response = await run_endpoint_function(
                   ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/bigluck/dev/.venv/lib/python3.12/site-packages/fastapi/routing.py", line 214, in run_endpoint_function
    return await run_in_threadpool(dependant.call, **values)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/bigluck/dev/.venv/lib/python3.12/site-packages/starlette/concurrency.py", line 39, in run_in_threadpool
    return await anyio.to_thread.run_sync(func, *args)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/bigluck/dev/.venv/lib/python3.12/site-packages/anyio/to_thread.py", line 56, in run_sync
    return await get_async_backend().run_sync_in_worker_thread(
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/bigluck/dev/.venv/lib/python3.12/site-packages/anyio/_backends/_asyncio.py", line 2177, in run_sync_in_worker_thread
    return await future
           ^^^^^^^^^^^^
  File "/Users/bigluck/dev/.venv/lib/python3.12/site-packages/anyio/_backends/_asyncio.py", line 859, in run
    result = context.run(func, *args)
             ^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/bigluck/dev/src/routes/route_create_query.py", line 304, in route_create_query
    raise e
  File "/Users/bigluck/dev/src/routes/route_create_query.py", line 296, in route_create_query
    cur.execute(res_query)
  File "/Users/bigluck/dev/.venv/lib/python3.12/site-packages/adbc_driver_manager/dbapi.py", line 698, in execute
    handle, self._rowcount = _blocking_call(
                             ^^^^^^^^^^^^^^^
  File "adbc_driver_manager/_lib.pyx", line 1546, in adbc_driver_manager._lib._blocking_call_impl
  File "adbc_driver_manager/_lib.pyx", line 1217, in adbc_driver_manager._lib.AdbcStatement.execute_query
  File "adbc_driver_manager/_lib.pyx", line 260, in adbc_driver_manager._lib.check_error
adbc_driver_manager.ProgrammingError: INVALID_ARGUMENT: [libpq] Failed to execute query 'SELECT n.oid,n.*,d.description FROM pg_catalog.pg_namespace n LEFT OUTER JOIN pg_catalog.pg_description d ON d.objoid=n.oid AND d.objsubid=0 AND d.classoid='pg_namespace'::regclass ORDER BY nspname': ERROR:  no binary output function available for type aclitem
. SQLSTATE: 42883

INFO:     127.0.0.1:59923 - "POST /v0/bi/query HTTP/1.1" 500 Internal Server Error

How can we reproduce the bug?

with (
    adbc_driver_postgresql.dbapi.connect(CONNECTION_URI) as conn,
    conn.cursor() as cur,
):
    # https://github.com/apache/arrow-adbc/issues/2093
    cur.adbc_statement.set_options(**{'adbc.postgresql.use_copy': 'false'})
    cur.execute("SELECT n.oid,n.*,d.description FROM pg_catalog.pg_namespace n LEFT OUTER JOIN pg_catalog.pg_description d ON d.objoid=n.oid AND d.objsubid=0 AND d.classoid='pg_namespace'::regclass ORDER BY nspname")

Environment/Setup

Postgresql 15.5
Python 3.12.3
adbc-driver-postgresql 1.4.0
pyarrow 19.0.1

[lidavidm]

https://www.postgresql.org/message-id/CAAvxfHdpbCjJ6VO%2BBmJgMVA1yu50xBAEGw2NWaNnug-W0rBHCg%40mail.gmail.com

I think it's at least somewhat intentional, to have a core type
that has no binary I/O so that that case can be tested.

That's rather annoying. Hmm.

[bigluck]

Yeah, I found it too.
My case is not a regular one. I'm not using the library to import/export data into a PostgreSQL database but to create empty PG tables starting from a list of arrow schemas (it's a weird case, I know) and to fetch some PG-related results (like the query above).
So I'm more than happy if I can turn off the binary mode support used by the library.

[lidavidm]

For the fetching, is the Arrow support important or would just plain ol' psycopg work?

[bigluck]

Ideally, I would prefer to retrieve the results as an Arrow table. However, I am open to alternative solutions. The core challenge I face is that I need to execute a query within a temporary context immediately after creating tables and schemas, followed by a rollback to the original state. Essentially, I'm building an ephemeral execution context specifically for the input query.

[lidavidm]

Ah, so they really do need to be a single driver/connection.

[bigluck]

yes, unfortunately I have no other options

[lidavidm]

Unfortunately there's not a way to "turn off" the binary support since the only support is the binary support (we never built a text format parser!) So the only way to support this would be to build a separate set of text format parsers from scratch.

[paleolimbot]

Could we do a "text only" version where the output schema is always struct<string, string, string, ...> and just return the raw libpq text output? (The text output is how we issue our own calls to the catalog and everything is in theory using the same helper under the hood with the binary/text flag turned on/off).