Pusher stops pushing when it falls behind, with lots of DNSLookupFailed errors

[matrixbot]

This issue has been migrated from #7113.

---

Our pusher process got behind. A couple of hours after it started to catch up, almost all pushes started failing:

Inspection of logs showed that they were failing with DNSLookupFailed errors:

2020-03-19 11:11:27,083 - synapse.http.client - 281 - INFO - httppush.process-57923 - Sending request POST http://10.103.0.7/_matrix/push/v1/notify
2020-03-19 11:12:28,647 - synapse.http.client - 328 - INFO - httppush.process-57923 - Error sending request to  POST http://10.103.0.7/_matrix/push/v1/notify: DNSLookupError Couldn't find the hostname '10.103.0.7'
2020-03-19 11:12:28,647 - synapse.push.httppusher - 391 - WARNING - httppush.process-57923 - Failed to push event <redacted> to  <redacted>: <class 'twisted.internet.error.DNSLookupError'> DNS lookup failed: Couldn't find the hostname '10.103.0.7'.
2020-03-19 11:12:32,225 - synapse.push.httppusher - 276 - INFO - httppush.process-57923 - Push failed: delaying for 1s