Failed to handle PDU: NoneType is not iterable

[turt2live]

This appears to have happened when receiving a transaction (from matrix.org in this case), and did cause calls to get_missing_events which appears to be a more complicated way to get the event. It does appear to eventually have received the event.

Homeserver has been based off of develop with some additional patches that should not affect the problem here: develop...turt2live:travis/t2bot.io

homeserver_1 - 2018-09-16 03:40:03,006 - synapse.storage.events - 441 - INFO - Calculating state delta for room !redacted_MUyU:matrix.org
homeserver_1 - 2018-09-16 03:40:03,007 - synapse.util.async_helpers - 218 - INFO - Releasing linearizer lock 'fed_room_pdu' for key '!redacted_MUyU:matrix.org'
homeserver_1 - 2018-09-16 03:40:03,009 - synapse.federation.federation_server - 203 - ERROR - Failed to handle PDU $redacted_NfGx:matrix.org: Traceback (most recent call last):
  File "/home/matrix/.synapse/local/lib/python2.7/site-packages/twisted/internet/defer.py", line 501, in errback
    self._startRunCallbacks(fail)
  File "/home/matrix/.synapse/local/lib/python2.7/site-packages/twisted/internet/defer.py", line 568, in _startRunCallbacks
    self._runCallbacks()
  File "/home/matrix/.synapse/local/lib/python2.7/site-packages/twisted/internet/defer.py", line 654, in _runCallbacks
    current.result = callback(current.result, *args, **kw)
  File "/home/matrix/.synapse/local/lib/python2.7/site-packages/twisted/internet/defer.py", line 1475, in gotResult
    _inlineCallbacks(r, g, status)
--- <exception caught here> ---
  File "/home/matrix/.synapse/local/lib/python2.7/site-packages/synapse/federation/federation_server.py", line 192, in process_pdus_for_room
    origin, pdu
  File "/home/matrix/.synapse/local/lib/python2.7/site-packages/synapse/federation/federation_server.py", line 621, in _handle_received_pdu
    origin, pdu, get_missing=True, sent_to_us_directly=True,
  File "/home/matrix/.synapse/local/lib/python2.7/site-packages/synapse/handlers/federation.py", line 235, in on_receive_pdu
    origin, pdu, prevs, min_depth
  File "/home/matrix/.synapse/local/lib/python2.7/site-packages/synapse/handlers/federation.py", line 389, in _get_missing_events_for_pdu
    get_missing=False
  File "/home/matrix/.synapse/local/lib/python2.7/site-packages/synapse/handlers/federation.py", line 313, in on_receive_pdu
    auth_chain=auth_chain,
  File "/home/matrix/.synapse/local/lib/python2.7/site-packages/synapse/handlers/federation.py", line 473, in _process_received_pdu
    state=state,
  File "/home/matrix/.synapse/local/lib/python2.7/site-packages/synapse/handlers/federation.py", line 1465, in _handle_new_event
    six.reraise(tp, value, tb)
  File "/home/matrix/.synapse/local/lib/python2.7/site-packages/synapse/handlers/federation.py", line 1455, in _handle_new_event
    backfilled=backfilled,
  File "/home/matrix/.synapse/local/lib/python2.7/site-packages/twisted/internet/defer.py", line 1416, in _inlineCallbacks
    result = result.throwExceptionIntoGenerator(g)
  File "/home/matrix/.synapse/local/lib/python2.7/site-packages/twisted/python/failure.py", line 491, in throwExceptionIntoGenerator
    return g.throw(self.type, self.value, self.tb)
  File "/home/matrix/.synapse/local/lib/python2.7/site-packages/synapse/federation/federation_server.py", line 621, in _handle_received_pdu
    origin, pdu, get_missing=True, sent_to_us_directly=True,
  File "/home/matrix/.synapse/local/lib/python2.7/site-packages/twisted/internet/defer.py", line 1416, in _inlineCallbacks
    result = result.throwExceptionIntoGenerator(g)
  File "/home/matrix/.synapse/local/lib/python2.7/site-packages/twisted/python/failure.py", line 491, in throwExceptionIntoGenerator
    return g.throw(self.type, self.value, self.tb)
  File "/home/matrix/.synapse/local/lib/python2.7/site-packages/synapse/handlers/federation.py", line 235, in on_receive_pdu
    origin, pdu, prevs, min_depth
  File "/home/matrix/.synapse/local/lib/python2.7/site-packages/twisted/internet/defer.py", line 1416, in _inlineCallbacks
    result = result.throwExceptionIntoGenerator(g)
  File "/home/matrix/.synapse/local/lib/python2.7/site-packages/twisted/python/failure.py", line 491, in throwExceptionIntoGenerator
    return g.throw(self.type, self.value, self.tb)
  File "/home/matrix/.synapse/local/lib/python2.7/site-packages/synapse/handlers/federation.py", line 389, in _get_missing_events_for_pdu
    get_missing=False
  File "/home/matrix/.synapse/local/lib/python2.7/site-packages/twisted/internet/defer.py", line 1416, in _inlineCallbacks
    result = result.throwExceptionIntoGenerator(g)
  File "/home/matrix/.synapse/local/lib/python2.7/site-packages/twisted/python/failure.py", line 491, in throwExceptionIntoGenerator
    return g.throw(self.type, self.value, self.tb)
  File "/home/matrix/.synapse/local/lib/python2.7/site-packages/synapse/handlers/federation.py", line 313, in on_receive_pdu
    auth_chain=auth_chain,
  File "/home/matrix/.synapse/local/lib/python2.7/site-packages/twisted/internet/defer.py", line 1416, in _inlineCallbacks
    result = result.throwExceptionIntoGenerator(g)
  File "/home/matrix/.synapse/local/lib/python2.7/site-packages/twisted/python/failure.py", line 491, in throwExceptionIntoGenerator
    return g.throw(self.type, self.value, self.tb)
  File "/home/matrix/.synapse/local/lib/python2.7/site-packages/synapse/handlers/federation.py", line 473, in _process_received_pdu
    state=state,
  File "/home/matrix/.synapse/local/lib/python2.7/site-packages/twisted/internet/defer.py", line 1416, in _inlineCallbacks
    result = result.throwExceptionIntoGenerator(g)
  File "/home/matrix/.synapse/local/lib/python2.7/site-packages/twisted/python/failure.py", line 491, in throwExceptionIntoGenerator
    return g.throw(self.type, self.value, self.tb)
  File "/home/matrix/.synapse/local/lib/python2.7/site-packages/synapse/handlers/federation.py", line 1465, in _handle_new_event
    six.reraise(tp, value, tb)
  File "/home/matrix/.synapse/local/lib/python2.7/site-packages/synapse/handlers/federation.py", line 1455, in _handle_new_event
    backfilled=backfilled,
twisted.internet.defer.FirstError: FirstError[#0, [Failure instance: Traceback: <type 'exceptions.TypeError'>: 'NoneType' object is not iterable
/home/matrix/.synapse/local/lib/python2.7/site-packages/twisted/internet/defer.py:654:_runCallbacks
/home/matrix/.synapse/local/lib/python2.7/site-packages/twisted/internet/defer.py:1475:gotResult
/home/matrix/.synapse/local/lib/python2.7/site-packages/twisted/internet/defer.py:1416:_inlineCallbacks
/home/matrix/.synapse/local/lib/python2.7/site-packages/twisted/python/failure.py:491:throwExceptionIntoGenerator
--- <exception caught here> ---
/home/matrix/.synapse/local/lib/python2.7/site-packages/synapse/storage/events.py:154:handle_queue_loop
/home/matrix/.synapse/local/lib/python2.7/site-packages/twisted/internet/defer.py:1416:_inlineCallbacks
/home/matrix/.synapse/local/lib/python2.7/site-packages/twisted/python/failure.py:491:throwExceptionIntoGenerator
/home/matrix/.synapse/local/lib/python2.7/site-packages/synapse/storage/events.py:312:persisting_queue
/home/matrix/.synapse/local/lib/python2.7/site-packages/twisted/internet/defer.py:1416:_inlineCallbacks
/home/matrix/.synapse/local/lib/python2.7/site-packages/twisted/python/failure.py:491:throwExceptionIntoGenerator
/home/matrix/.synapse/local/lib/python2.7/site-packages/synapse/storage/events.py:195:f
/home/matrix/.synapse/local/lib/python2.7/site-packages/twisted/internet/defer.py:1418:_inlineCallbacks
/home/matrix/.synapse/local/lib/python2.7/site-packages/synapse/storage/events.py:453:_persist_events
]]
homeserver_1 - 2018-09-16 03:40:03,018 - synapse.handlers.federation - 226 - INFO - Acquiring lock for room '!redacted_MUyU:matrix.org' to fetch 1 missing events: [u'$redacted_NfGx:matrix.org']...
homeserver_1 - 2018-09-16 03:40:03,018 - synapse.util.async_helpers - 205 - INFO - Acquired uncontended linearizer lock 'fed_room_pdu' for key '!redacted_MUyU:matrix.org'

[turt2live]

This is still happening:

homeserver_1 - 2018-09-27 16:09:35,629 - synapse.federation.federation_server - 205 - ERROR - PUT-4013-$redacted_dvww:matrix.org - Failed to handle PDU $redacted_dvww:matrix.org: Traceback (most recent call last):
--- <exception caught here> ---
  File "/home/matrix/.synapse/local/lib/python2.7/site-packages/synapse/federation/federation_server.py", line 194, in process_pdus_for_room
    origin, pdu
  File "/home/matrix/.synapse/local/lib/python2.7/site-packages/synapse/federation/federation_server.py", line 623, in _handle_received_pdu
    origin, pdu, sent_to_us_directly=True,
  File "/home/matrix/.synapse/local/lib/python2.7/site-packages/synapse/handlers/federation.py", line 268, in on_receive_pdu
    origin, pdu, prevs, min_depth
  File "/home/matrix/.synapse/local/lib/python2.7/site-packages/synapse/handlers/federation.py", line 548, in _get_missing_events_for_pdu
    sent_to_us_directly=False,
  File "/home/matrix/.synapse/local/lib/python2.7/site-packages/synapse/handlers/federation.py", line 438, in on_receive_pdu
    auth_chain=auth_chain,
  File "/home/matrix/.synapse/local/lib/python2.7/site-packages/synapse/handlers/federation.py", line 637, in _process_received_pdu
    yield self._handle_new_events(origin, event_infos)
  File "/home/matrix/.synapse/local/lib/python2.7/site-packages/synapse/handlers/federation.py", line 1672, in _handle_new_events
    backfilled=backfilled,
  File "/home/matrix/.synapse/local/lib/python2.7/site-packages/synapse/handlers/federation.py", line 2531, in persist_events_and_notify
    backfilled=backfilled,
  File "/home/matrix/.synapse/local/lib/python2.7/site-packages/synapse/storage/events.py", line 273, in persist_events
    defer.gatherResults(deferreds, consumeErrors=True)
twisted.internet.defer.FirstError: FirstError[#0, [Failure instance: Traceback: <type 'exceptions.TypeError'>: 'NoneType' object is not iterable
/home/matrix/.synapse/local/lib/python2.7/site-packages/twisted/internet/defer.py:654:_runCallbacks
/home/matrix/.synapse/local/lib/python2.7/site-packages/twisted/internet/defer.py:1475:gotResult
/home/matrix/.synapse/local/lib/python2.7/site-packages/twisted/internet/defer.py:1416:_inlineCallbacks
/home/matrix/.synapse/local/lib/python2.7/site-packages/twisted/python/failure.py:491:throwExceptionIntoGenerator
--- <exception caught here> ---
/home/matrix/.synapse/local/lib/python2.7/site-packages/synapse/storage/events.py:154:handle_queue_loop
/home/matrix/.synapse/local/lib/python2.7/site-packages/twisted/internet/defer.py:1416:_inlineCallbacks
/home/matrix/.synapse/local/lib/python2.7/site-packages/twisted/python/failure.py:491:throwExceptionIntoGenerator
/home/matrix/.synapse/local/lib/python2.7/site-packages/synapse/storage/events.py:312:persisting_queue
/home/matrix/.synapse/local/lib/python2.7/site-packages/twisted/internet/defer.py:1416:_inlineCallbacks
/home/matrix/.synapse/local/lib/python2.7/site-packages/twisted/python/failure.py:491:throwExceptionIntoGenerator
/home/matrix/.synapse/local/lib/python2.7/site-packages/synapse/storage/events.py:195:f
/home/matrix/.synapse/local/lib/python2.7/site-packages/twisted/internet/defer.py:1418:_inlineCallbacks
/home/matrix/.synapse/local/lib/python2.7/site-packages/synapse/storage/events.py:453:_persist_events
]]

[richvdh]

This was introduced by #3595, though it looks like there has been some brokenness in this area since at least #1836.

[richvdh]

fixed via #3995