Synapse fails to make DNS query for its own hostname when attempting invite via 3PID #9475
Comments
|
Relevant links: Btw, what twisted version are you using? ( |
|
this seems to be https://twistedmatrix.com/trac/ticket/9691 |
|
I'm surprised though. I didn't think we used I've been very confused about this in the past though (see #7113 (comment) and https://github.com/matrix-org/synapse/blob/v1.27.0/synapse/app/_base.py#L393), and it's possible I'm still confused. It seems like somehow |
|
I'm running with docker as per http://github.com/spantaleev/matrix-docker-ansible-deploy
|
|
@alex-caelus do you have a |
|
I think this needs more info, like synapse version, OS version, and maybe the DNS server's logs denoting the |
clokep
added
the
X-Needs-Info
|
It's a clean and up-to-date Ubuntu 20.04. Matrix was installed using the ansible scripts from https://github.com/spantaleev/matrix-docker-ansible-deploy. Which means docker images. https://paste.ubuntu.com/p/m5DSsWqy33/ But this is wierd: Perhaps the title of this issue is wrong. I tried to reproduce the stacktrace on my own before I filed this bug. I ran a few scripts on my desktop (tried both windows and linux) with twisted and I got the same exception and then I used wireshark to look at the traffic. Which is why I came to the conclusion in my first post. However, when I do a tcpdump on the matrix server instead I get the correct requests for A and AAAA records. Which means the stacktrace in my first post is caused by something else. I'm attaching the tcpdump, in case you are interested but I can see nothing wrong in it. (Note the bind9 man-in-the-middle: 127.0.0.1 <-> 127.0.0.53 <-> 10.0.0.1). |
|
I'll take a look at that once i have some time, thanks for coming through! |
|
@alex-caelus i'm not seeing any |
isn't that exactly what they said in #9475 (comment) ? |
Well, my assumed root-cause might be wrong so the title is off. But the backtrace and symptoms (failed to invite via email-address) still affect me, so yes the bug is still persisting. |
|
Alright, just for clarification, does your DNS server still register |
|
No, it apparently never did. I just assumed that it did (see my earlier comment) |
alex-caelus
changed the title
|
I edited title and description, it was confusing, to say the least. |
|
Then the question becomes; why is it failing a lookup to that address (from the server)? Could you maybe do |
|
If you look at the pcap file I attached you can see that the query is successful but still the exception happens. Here is a screenshot of the relevant parts, for those that do not have wireshark installed:
|
|
I see three possibilities myself, based on the pcap:
I have trouble believing in number 2 since I have had no other issues with other programs or devices on the same network before.
Sure thing: |
|
oh whoops, forgot thanks for the information, though, this is probably enough to figure out this bug |
|
Hey, I have the same problem:
also it runs in container, I have no issue getting the ip on the host |
|
Hey, I am also experiencing this issue:
I am running the latest Synapse Docker Container A few points that might be interesting:
|
|
Ok, after an additional digging through the code and the logs, I found the solution to my problem:
After setting the blacklist and excluding my IP range, name resolution (and federation) started working. Strangely, it was not enough to put the IP range on the whitelist, I had to remove it from the blacklist. |
|
@evodicka the problem with the whitelist is likely due to #10115 @alex-caelus looks like your server is trying to send a request to a host with a blacklist IP (from your pcap the answer to the DNS query is in 10.0.*, which is private and blacklisted by default). Is it trying to talk to itself? If so that's a bug, as Synapse should never attempt to talk to itself over federation. |
|
I don't know about actually talking to, but yes. It's trying to resolve it's own hostname. I'm inviting |
|
Here's the synapse logs again (v1.35.1): |
|
Does |
|
it resolves to |
|
Ok, yup, that will be in the IP blacklist by default. Looks like you're something on I wonder if we should log loudly every time we blacklist an IP or something? |
|
I'm using https://github.com/spantaleev/matrix-docker-ansible-deploy to deploy everything on the same server, including both synapse and ma1sd. But why is it trying to look up itself? What is special about my setup so that not everyone is having the same issue? Also, should I remove my ip from the blacklist or is it there for a reason? |
See synapse/synapse/http/client.py Lines 184 to 186 in 64887f0
You're running two matrix services that are attempting to communicate via a private IP, which is not common, although it seems I've been fielding a bunch of questions about this frequently. |
#4857 is probably related here |
|
stumbled upon this issue as well. My setup is based on https://github.com/spantaleev/matrix-docker-ansible-deploy, too. It's a little bit weird to have synapse refusing to make DNS queries against our own internal nameservers and if I look at the sample config file synapse/docs/sample_config.yaml Line 204 in e9f2ad8 I fail to see DNS queries being mentioned here. So IMHO at least DNS queries should be mentioned there as well plus any other resources that might fail, too. |
DNS queries are (generally) made using |
Maybe I'm dense, but that sounds like synapse is not blocking the dns queries based on the blacklist. I'm getting a bit confused here |
Well, it looks like Synapse is not blocking the DNS queries, but it throws away the resolved IPs if they are on the blacklist. You can checkout client.py to have a look what it actually does. |
allright, but if I read the stack trace correctly, this has got nothing to do with client.py (as a sidenote, client.py even nicely logs if it throws away an IP because it has been blacklisted: synapse/synapse/http/client.py Lines 178 to 181 in 64887f0 The stack trace reads like this (copied from comment 861258334) And if you look at the actual error logged, it says So the message explicitly says that the DNS lookup failed. That's definitely something that should be corrected to say that the DNS response has been filtered out due to the blacklist. |
Yes, it does. And when I ran into this issue, I could actually find exactly that log a couple of lines above the stacktrace. I just wasn't paying attention to it at first, because I focused only on the stacktrace.
I am not an expert on this issue, but what I understood when investigating that issue after I ran into it:
And yes, I find that very misleading. |
erikjohnston
added
T-Task
|
I've opened #10224 to track the fact the stack trace is quite misleading. I think otherwise all the issues that have been brought up in this thread have been resolved? |
|
I'm not sure & I'm no expert here. From my POV, I still see 3 potential remaining problems:
I think problem one has been tracked here #4857 IMHO, the second problem really should just never occur. I fail to see any (security) implications when a service like Synapse is allowed to talk to itself. From my POV, this seems to be an almost "natural" behavior :). No default setting of a blacklist should disallow this. And per the third potential problem, I honestly don't understand why Synapse should not be allowed to talk to internal networks at all, or at least to servers that share its same network segment. So all I can say is that problems should be fixed where they occur, but I am in no position to say what the "right place" for such a fix is. |
That is true, but it looks to me from the stack traces that its talking to a configured identity server, which happens to have the same domain as the home server implementation.
There are a number of ways in which clients and other servers can get Synapse to make requests to arbitrary IPs and ports, completely bypassing any firewalls etc. This can easily lead to security issues if people don't realise that that is possible (and generally its rare for there to be any reason for Synapse to talk to private IPs). |
|
Nobody covered in this thread how to remove an ip range from the blacklist. I am running HAProxy on pfsense and it is handling request externally just fine, but internally I use a virtual IP so that clients on the local network also get routed, (I do not use NAT reflection) Unfortunately the result is: Update.sydent gives you a lot of config values under [default] you cannot just edit these, you have to figure out which category they belong to below, once I set the whitelist below the [general] section i am now past that hurdle and onto the next one. sydent.conf: |
@Jieiku your Homeserver is separate from Sydent (identify server). The homeserver's black- and whitelists are separate to the identity server's black- and whitelists. I strongly recommend you do not set the ip.blacklist as above, because doing so wipes out the defaults, which is probably not what you want. Instead, you want to set If that doesn't resolve the problem, please open an issue in the Sydent repo, or ask in the matrix room |
|
Thank you! I appreciate it! I did actually figure out that it was the sydent config to edit. The confusing part of this config is that there is a [default] section, but changing stuff there does not actually make any changes. you have to add the lines to the categories below such as [general] I think sydent config file should have the sections below populated, and just comment out the lines. It was not at all obvious how to edit this file. |
Hi!
Synapse makes DNS ANY (ALL) requests through use of twisted, which sometimes fail.EDIT: My bad, this was apparently not what was happening, see thread for discussion.For example when I'm trying to invite '[email protected]' as per instructions on https://github.com/ma1uta/ma1sd/blob/master/docs/getting-started.md the invitation fails. Upon investigation I see the following in the logs:
Note that it's trying to make a DNS lookup to my own server, on my own network which my local dns server responds to.
Unfortunately the dns request is of type 255 or ANY/ALL which has been deprecated for many years. My DNS server, correctly, returns an empty response.BTW, the following code also fails (for me) when doing a lookup on matrix.org, because google's DNS server responds with a HINFO instead of a A or AAA record (or CNAME):
The text was updated successfully, but these errors were encountered: