Federation DNS Lookup fails for healthy domain

[aryasenna]

Hi,

I am getting significant amount of DNS lookup failures message in my Synapse log.

(excerpt only for readability reason)

2019-08-08 11:21:41,212 - synapse.http.matrixfederationclient - 485 - WARNING - PUT-7679- {POST-O-1905} [matrix.org] Request failed: POST matrix://matrix.org/_matrix/key/v2/query: DNSLookupError("Couldn't find the hostname 'matrix.org.cdn.cloudflare.net'",)
2019-08-08 11:21:41,520 - synapse.http.matrixfederationclient - 485 - WARNING - PUT-7649- {POST-O-1904} [matrix.org] Request failed: POST matrix://matrix.org/_matrix/key/v2/query: DNSLookupError("Couldn't find the hostname 'matrix.org.cdn.cloudflare.net'",)
2019-08-08 11:21:43,668 - synapse.http.matrixfederationclient - 485 - WARNING - PUT-7697- {POST-O-1906} [matrix.org] Request failed: POST matrix://matrix.org/_matrix/key/v2/query: DNSLookupError("Couldn't find the hostname 'matrix.org.cdn.cloudflare.net'",)
2019-08-08 11:21:44,043 - synapse.http.matrixfederationclient - 485 - WARNING - PUT-7708- {POST-O-1907} [matrix.org] Request failed: POST matrix://matrix.org/_matrix/key/v2/query: DNSLookupError("Couldn't find the hostname 'matrix.org.cdn.cloudflare.net'",)
2019-08-08 11:21:46,534 - synapse.http.matrixfederationclient - 485 - WARNING - PUT-7128- {POST-O-1908} [matrix.org] Request failed: POST matrix://matrix.org/_matrix/key/v2/query: DNSLookupError("Couldn't find the hostname 'matrix.org.cdn.cloudflare.net'",)
2019-08-08 11:21:46,581 - synapse.http.matrixfederationclient - 485 - WARNING - PUT-7435- {POST-O-1909} [matrix.org] Request failed: POST matrix://matrix.org/_matrix/key/v2/query: DNSLookupError("Couldn't find the hostname 'matrix.org.cdn.cloudflare.net'",)
2019-08-08 11:21:47,011 - synapse.http.matrixfederationclient - 485 - WARNING - PUT-7347- {POST-O-1910} [matrix.org] Request failed: POST matrix://matrix.org/_matrix/key/v2/query: DNSLookupError("Couldn't find the hostname 'matrix.org.cdn.cloudflare.net'",)
2019-08-08 11:21:47,123 - synapse.http.matrixfederationclient - 485 - WARNING - PUT-7344- {POST-O-1911} [matrix.org] Request failed: POST matrix://matrix.org/_matrix/key/v2/query: DNSLookupError("Couldn't find the hostname 'matrix.org.cdn.cloudflare.net'",)

Matrix.org can be resolved correctly on the host system:

user@host:~$ nslookup matrix.org.cdn.cloudflare.net
Server:		8.8.8.8
Address:	8.8.8.8#53

Non-authoritative answer:
Name:	matrix.org.cdn.cloudflare.net
Address: 104.20.20.236
Name:	matrix.org.cdn.cloudflare.net
Address: 104.20.21.236
Name:	matrix.org.cdn.cloudflare.net
Address: 2606:4700:10::6814:15ec
Name:	matrix.org.cdn.cloudflare.net
Address: 2606:4700:10::6814:14ec

user@host~$ host matrix.org.cdn.cloudflare.net
matrix.org.cdn.cloudflare.net has address 104.20.21.236
matrix.org.cdn.cloudflare.net has address 104.20.20.236
matrix.org.cdn.cloudflare.net has IPv6 address 2606:4700:10::6814:15ec
matrix.org.cdn.cloudflare.net has IPv6 address 2606:4700:10::6814:14ec
matrix.org.cdn.cloudflare.net mail is handled by 10 polemos.matrix.org.
matrix.org.cdn.cloudflare.net mail is handled by 20 zelus.matrix.org.
user@host:~$ nslookup bau-ha.us
Server:		8.8.8.8
Address:	8.8.8.8#53

I am not sure if Synapse uses twisted.names for resolving hostname to IP address,
Issue #1002 seem to indicate so, but @hawkowl at #synapse-dev room said this is not the case.

If twisted.names is indeed used, there is some weird behavior I noticed by which gethostbyname() refuses to work with combination of some domain and resolver (https://twistedmatrix.com/trac/ticket/9691).

Is this even reproducible on other HS setup?

[richvdh]

synapse does not (normally) use twisted.names (see also #5053).

I think this must be specific to your setup, given the number of people that federate with matrix.org and have never reported this. What does your /etc/resolv.conf contain?

[richvdh]

It seems it's more complicated and sometimes twisted.names is used.

but @hawkowl at #synapse-dev room said this is not the case.

That isn't what she said there.

Nevertheless I'd be surprised if this was https://twistedmatrix.com/trac/ticket/9691, and something specific to your environment seems more likely.

[aryasenna]

My resolve.conf content was simply

nameserver 1.1.1.1
nameserver 1.0.0.1

I linked the matrix room chat, she said Synapse uses Python getbyaddress() or something, instead of twisted's implementation.

I found out that I am getting nslookup timeout error when I do repeated nslookup. (Maybe my provider rate-limits the DNS outbound query as some sort of DDoS mitigation).

In any case, Synapse certainly performs repeated lookup due to federation. So I installed systemd-resolved which caches the lookup result, and I have no more problems.

Probably it is specific to my setup, but Synapse's failure mode when dns lookup is rate-limited is not well defined here.

I'd think Synapse might want to rate limit the DNS query or cache it.

[richvdh]

I linked the matrix room chat, she said Synapse uses Python getbyaddress() or something, instead of twisted implementation.

No, what she actually said was:

we should be using getaddrinfo by default

that is an aspiration, not a statement of fact. In practice twisted.names is used sometimes.

I'd think Synapse might want to rate limit the DNS query or cache it.

It can't ratelimit DNS without ratelimiting outgoing requests. That's basially #5373.

Caching is something that should be done by your system, not synapse. Installing systemd-resolved is a valid solution.