Database exception: can't adapt type 'dict'

[ShadowJonathan]

Traceback

synapse_1  | 2020-10-06 18:27:29,810 - synapse.handlers.federation - 415 - WARNING - PUT-104-$QPiPwGPep1u_EzDQJn6l55K7NQLY8vk-qZ_C7jJzEik-$lRU96CKrIrQZRWyju9NQepoU4Q-9PCCcD5d05__HOv0 - [!EgoVrPyOnPSVXERlxj:gruenhage.xyz $lRU96CKrIrQZRWyju9NQepoU4Q-9PCCcD5d05__HOv0] Error attempting to resolve state at missing prev_events
synapse_1  | Traceback (most recent call last):
synapse_1  |   File "/usr/local/lib/python3.7/site-packages/synapse/handlers/federation.py", line 375, in on_receive_pdu
synapse_1  |     origin, room_id, p, include_event_in_state=True
synapse_1  |   File "/usr/local/lib/python3.7/site-packages/synapse/handlers/federation.py", line 591, in _get_state_for_room
synapse_1  |     destination, room_id, desired_events
synapse_1  |   File "/usr/local/lib/python3.7/site-packages/synapse/handlers/federation.py", line 649, in _get_events_from_store_or_dest
synapse_1  |     destination=destination, room_id=room_id, events=missing_events
synapse_1  |   File "/usr/local/lib/python3.7/site-packages/synapse/handlers/federation.py", line 1268, in _get_events_and_persist
synapse_1  |     destination, event_infos,
synapse_1  |   File "/usr/local/lib/python3.7/site-packages/synapse/handlers/federation.py", line 2000, in _handle_new_events
synapse_1  |     backfilled=backfilled,
synapse_1  |   File "/usr/local/lib/python3.7/site-packages/synapse/handlers/federation.py", line 2976, in persist_events_and_notify
synapse_1  |     event_and_contexts, backfilled=backfilled
synapse_1  |   File "/usr/local/lib/python3.7/site-packages/synapse/storage/persist_events.py", line 226, in persist_events
synapse_1  |     defer.gatherResults(deferreds, consumeErrors=True)
synapse_1  | twisted.internet.defer.FirstError: FirstError[#0, [Failure instance: Traceback: <class 'psycopg2.ProgrammingError'>: can't adapt type 'dict'
synapse_1  | /usr/local/lib/python3.7/site-packages/twisted/internet/defer.py:654:_runCallbacks
synapse_1  | /usr/local/lib/python3.7/site-packages/twisted/internet/defer.py:1475:gotResult
synapse_1  | /usr/local/lib/python3.7/site-packages/twisted/internet/defer.py:1416:_inlineCallbacks
synapse_1  | /usr/local/lib/python3.7/site-packages/twisted/python/failure.py:512:throwExceptionIntoGenerator
synapse_1  | --- <exception caught here> ---
synapse_1  | /usr/local/lib/python3.7/site-packages/synapse/storage/persist_events.py:148:handle_queue_loop
synapse_1  | /usr/local/lib/python3.7/site-packages/synapse/storage/persist_events.py:254:persisting_queue
synapse_1  | /usr/local/lib/python3.7/site-packages/synapse/storage/persist_events.py:430:_persist_events
synapse_1  | /usr/local/lib/python3.7/site-packages/synapse/storage/databases/main/events.py:174:_persist_events_and_state_updates
synapse_1  | /usr/local/lib/python3.7/site-packages/synapse/storage/database.py:541:runInteraction
synapse_1  | /usr/local/lib/python3.7/site-packages/synapse/storage/database.py:590:runWithConnection
synapse_1  | /usr/local/lib/python3.7/site-packages/twisted/python/threadpool.py:250:inContext
synapse_1  | /usr/local/lib/python3.7/site-packages/twisted/python/threadpool.py:266:<lambda>
synapse_1  | /usr/local/lib/python3.7/site-packages/twisted/python/context.py:122:callWithContext
synapse_1  | /usr/local/lib/python3.7/site-packages/twisted/python/context.py:85:callWithContext
synapse_1  | /usr/local/lib/python3.7/site-packages/twisted/enterprise/adbapi.py:306:_runWithConnection
synapse_1  | /usr/local/lib/python3.7/site-packages/twisted/python/compat.py:464:reraise
synapse_1  | /usr/local/lib/python3.7/site-packages/twisted/enterprise/adbapi.py:297:_runWithConnection
synapse_1  | /usr/local/lib/python3.7/site-packages/synapse/storage/database.py:587:inner_func
synapse_1  | /usr/local/lib/python3.7/site-packages/synapse/storage/database.py:429:new_transaction
synapse_1  | /usr/local/lib/python3.7/site-packages/synapse/logging/utils.py:71:wrapped
synapse_1  | /usr/local/lib/python3.7/site-packages/synapse/storage/databases/main/events.py:404:_persist_events_txn
synapse_1  | /usr/local/lib/python3.7/site-packages/synapse/storage/databases/main/events.py:942:_update_metadata_tables_txn
synapse_1  | /usr/local/lib/python3.7/site-packages/synapse/storage/databases/main/events.py:1111:_store_room_members_txn
synapse_1  | /usr/local/lib/python3.7/site-packages/synapse/storage/database.py:741:simple_insert_many_txn
synapse_1  | /usr/local/lib/python3.7/site-packages/synapse/storage/database.py:215:executemany
synapse_1  | /usr/local/lib/python3.7/site-packages/synapse/storage/database.py:238:_do_execute
synapse_1  | ]]

This happened after a server restart, when booting and grabbing events from servers (probably).

Version information

• Version: 1.20.1

• Install method: Docker Image

• Platform: Debian 10

[clokep]

This might fixed by ##8385? I'm not sure any of the other items there could cause this.

[ShadowJonathan]

Does 1.20.1 have eb611ec? (yes)

I've had the problem that led to #8385 before, and it was fixed after that PR was brought in, so i'm doubtful this is covered by that.

[clokep]

Does 1.20.1 have eb611ec? (yes)

What? This commit doesn't even end up in the history since we squash merge. According to the changelog, #8385 was included in v1.21.0rc1.

I've had the problem that led to #8385 before, and it was fixed after that PR was brought in, so i'm doubtful this is covered by that.

That PR hasn't been released yet. 😕

[ShadowJonathan]

Oh whoops, I didn't check the right branch then.

[richvdh]

assuming this is fixed