Reverse proxy causes "Invalid signature for server ..." resulting in inability to accept invites #3294
Comments
|
Removing the proxy on the federation port clears the issue, so it seems related to this |
|
I have the same issue.
Synapse receives request from Request content (as Python string): The same JSON object pretty printed with indentation: JSON object passed to The same JSON object pretty printed: Arguments passed as a result to
pk (public key), represented as Python string: Result: |
|
Another interesting finding. |
|
Another update. |
|
After I've upgraded another server to 0.30.0 (from my OBS), invitations stopped working. |
|
There's also another report of signature check failures at https://riot.im/develop/#/room/#matrix-dev:matrix.org/$15273484311ZxXNG:theta.eu.org |
|
By adding debug print statements to another server, I've found what causes discrepancy between sent and received data. The difference is in URL. Another server (federation client) sends the following URL: But the server itself receives the following URL: Note that URL being sent has Quoting URL for sending (and signing JSON with it) was introduced in dab87b8 of #3082 and this change was included in v0.27.3. It's interesting to note that Apache HTTPD logs show quoted URL. So, it's still possible that it's Investigating further... |
|
Great success! Yes, it was (please ignore Problem solved! A note should be added to "Using a reverse proxy with Synapse" section of Synapse |
|
Thanks a lot @OlegGirko I am using nginx and I had the same problem. Here is my previous configuration: I changed the The nginx proxy_pass documentation states that without a path after the server in the |
|
@remjey your mentioned fix is working fine for me. 👍 I believe this to be a bug, my understanding of the RFCs is that at least the |
richvdh
changed the title
richvdh
changed the title
|
Sigh. The right solution is probably to closely specify how URL should be encoded, and ensure that is done at both ends before signing/verifying. For now, we should probably just add a warning to the list of things to be careful of when reverse-proxying federation: https://github.com/matrix-org/synapse/blob/master/README.rst#reverse-proxying-the-federation-port. |
|
May be inserting something like: json_object['uri'] = '/'.join(
tuple(
urllib.quote(urllib.unquote(arg), safe='') for arg in json_object.get('uri').split('/')
)
)before line https://github.com/matrix-org/synapse/blob/develop/synapse/crypto/keyring.py#L784 completely solve the problem? |
|
Small fix - do not touch query parameters: parts = urlparse.urlsplit(json_object.get('uri'))
path = '/'.join(
tuple(
urllib.quote(urllib.unquote(arg), safe='') for arg in parts[2].split('/')
)
)
json_object['uri'] = urlparse.urlunparse(('', '', path, '', parts[3], '')) |
|
This is a thing that is still a problem, an we really need to take the time to figure out properly. |
|
Also got hit by this in v0.99 with apache2 acting as a reverse-proxy on the federation port. Stock apache2 config from debian stretch and this was my virtualhost: synapse was configured with disabled TLS on the federation port, but no certificate config was changed. Switching to nginx worked. |
|
@maxidorius you didn't by any chance check if nocanon would have worked with apache? |
|
I've already checked (see my comments from May 27 2018 above) and I can ensure that it won't work without |
|
@ptman I did not check, no. |
|
@ptman, @maxidorius: |
|
@OlegGirko yes your comments are for port 8008, which is in the recommended setup for client API only and reverse proxy was NOT recommended for federation port (8448) until v0.99 of synapse. v0.99 now accepts regular certificates. My comment was specifically targeted at v0.99 which supports reverse proxy as per synapse devs themselves. My comments was therefore that stock Apache2, on federation port (8448) does not work out of the box. Whichever way that ends up being translated in terms of documentation is up to them. I am merely reporting the issue(s) I'm running into and a regular user expectation that if it works on C2S, it should work on S2S easily. That it doesn't is either a lack of clear documentation about it, or possibly a bug on how the path is dealt with when checking signatures. The spec doc doesn't talk about encoding in the path, so I guess that can lead to signatures errors anyway. |
|
@maxidorius:
And it should not. If you want to use reverse proxy, you have to configure it first. And using There is no expectation that if There is no need to report this issue again. The case is closed long ago, solution is well-known. I think, the only reason why this issue is still open is that there is no clear long-term resolution: whether it's enough to leave a requirement to use |
|
Please guys, be nice to each other and take the time to understand what others are saying. There's no need to get angry about it. I think the situation is fairly clear:
To solve the problem, we either need to make it very clear in our documentation that you need to use |
|
@richvdh Thank you for summarizing my feedback nicely. |
The word "unreliable" doesn't sound right here. It means "sometimes working, sometimes not", but in reality it's deep in "not working at all" territory. Messages still pass through, but invitation requests don't pass at all.
But there is no time for deliberation whether to use first or second approach. First approach should be implemented immediately right now, and requirement to disable URL canonicalisation should be so prominent in documentation that we see no further confusion about it. I think this should also be mentioned in upgrade instructions to prevent this problem appearing in case someone decides to reuse reverse proxy configuration that was used for client-only requests before. As the long-term solution, I would also prefer if Synapse was converting URLs to its own canonicalised representation for federation before signing and checking for signature correctness. This will not only lift the requirement for disabling URL canonicalisation by reverse proxy, but also prevent possible problems in case there is some non-reverse (probably transparent) proxy between federating homeservers. However, I don't think that it's realistic to expect this solution to be implemented quickly enough to not bother with first (documentation-based) solution. |
|
Having looked at this a little more: It appears that apache will 404 any request with a %2F in the URL unless I haven't dug into the nginx case particularly, but fewer people seem to be getting tripped up by that. I'll try to fix up the documentation this week. |
|
Isn't it a violation of HTTP protocol to treat for I have a feeling that allowing anything that can have I understand that when federation protocol was originally designed, nobody envisioned usernames like |
|
Could this issue be re-opened? It would be nice if this apache configuration fragility could be avoided. I just spend the better part of a day wrestling with this before realizing it wasn't my code or the Nio library, but a completely random server misconfiguration (on a server which I don't control). Apparently a fix is technically possible? #3611 |
|
Making this less fragile is a protocol issue, not a synapse one. It is already tracked at matrix-org/matrix-spec#561. |
and others
Description
I am unable to invite people on some servers into a chat.
In Riot I see no difference (but no one ever accepts the invite ;) )
Synapse always logs something similiar to this:
The remote side receives the invite but is unable to accept it ("unauthorized").
Federation tester did not show any issues at the time of the error for both involved servers:
https://matrix.org/federationtester/api/report?server_name=utzutzutz.net
https://matrix.org/federationtester/api/report?server_name=asra.gr
Several synapse operators seem to have the same issue. It does not seem to be related to OS or installation method.
See https://matrix.to/#/!HsxjoYRFsDtWBgDQPh:matrix.org/$1527262498154duwWE:fws.fr for some more examples.
Steps to reproduce
Version information
If not matrix.org:
The text was updated successfully, but these errors were encountered: