"This is not a valid Matrix server Address" on trying to leave public room over federation

[mpatey]

I joined a public room on another homeserver to test federation. That was a week or so ago because I forgot about it. Now when I try to leave the room I can't do it.

On element on Android it says: This is not a valid Matrix server Address
On element on Windows and on the web (app.element.io) it says Error leaving room Unexpected server error trying to leave the room

In the server logs I can't see any error messages. On #synapse:matrix.org it was suggested I clear the app cache, but this did not fix it.

I don't know if it is relevant, but the room I am stuck in has version 6, whereas rooms I create on the server are version 5.

I am running matrix-synapse 1.19.1-1 on debian

[anoadragon453]

Synapse should be able to leave a room without the help of a remote server. If the remote server is down, then it's best effort in terms of alerting it to your attempt to leave - but should not block it.

You mentioned being unable to see an errors in your server logs. Can you try making sure you're at least on INFO logging (not WARNING), and try searching case-sensitively for ERROR and Exception.

If that doesn't turn up anything, then some debug logs from your client would be helpful as well. You can send these from each app's settings, or by physically shaking your mobile device (if you have that option turned on).

[richvdh]

Can you try making sure you're at least on INFO logging (not WARNING), and try searching case-sensitively for ERROR and Exception.

ERRORs will get logged whether or not you reduce your log level to INFO...

[mpatey]

I have reproduced the problem on the Android element client and immediately sent a bug report. Have you got it?

Here is a part of the homeserver.log from the server at the same time (I have overwritten my server name with *). It is all INFO level messages and as I said nothing obvious to me:

`2020-09-28 18:44:09,976 - synapse.access.http.8008 - 311 - INFO - GET-1010- ::1 - 8008 - {@matt:**********} Processed request: 23.003sec/0.007sec (0.017sec, 0.000sec) (0.000sec/0.000sec/0) 195B 200 "GET /_matrix/client/r0/sync?filter=1&timeout=30000&since=s8533_21152_0_506_492_1_245_291_1 HTTP/1.1" "Element (Riot.im)/1.0.7 (Linux; U; Android 10; Aquaris X Pro Build/QQ3A.200805.00; Flavour GooglePlay; MatrixAndroidSDK_X 0.0.1)" [0 dbevts]

2020-09-28 18:44:09,997 - synapse.access.http.8008 - 311 - INFO - GET-1007- ::1 - 8008 - {@matt:**********} Processed request: 30.027sec/0.024sec (0.023sec, 0.000sec) (0.000sec/0.000sec/0) 195B 200 "GET /_matrix/client/r0/sync?filter=1&timeout=30000&since=s8533_21152_0_506_492_1_245_291_1 HTTP/1.1" "Element (Riot.im)/1.0.7 (Linux; U; Android 10; Aquaris X Pro Build/QQ3A.200805.00; Flavour GooglePlay; MatrixAndroidSDK_X 0.0.1)" [0 dbevts]
2020-09-28 18:44:20,333 - synapse.handlers.presence - 348 - INFO - persist_presence_changes-1609- Persisting 1 unpersisted presence updates

2020-09-28 18:44:40,276 - synapse.access.http.8008 - 311 - INFO - GET-1011- ::1 - 8008 - {@matt:**********} Processed request: 30.024sec/0.003sec (0.015sec, 0.005sec) (0.000sec/0.000sec/0) 269B 200 "GET /_matrix/client/r0/sync?filter=1&timeout=30000&since=s8533_21152_0_506_492_1_245_291_1 HTTP/1.1" "Element (Riot.im)/1.0.7 (Linux; U; Android 10; Aquaris X Pro Build/QQ3A.200805.00; Flavour GooglePlay; MatrixAndroidSDK_X 0.0.1)" [0 dbevts]`

[anoadragon453]

From the logs I can see that you're getting a fair amount of 404s from your reverse proxy server. Can you check the error logs of your reverse proxy and see which URLs are failing to reach your homeserver? I'm guessing that some of them may be related to room joins, and this is why we're not seeing any errors in the homeserver's logs.

[mpatey]

Here is a section of the access.log from apache for the same time period as the other logs. I **d out the remote homeserver (not sure whether that's necessary or not). There are a couple of 404s for leaving the room. So I simply have a problem with the reverse proxy?

`79.72.35.240 - - [28/Sep/2020:18:43:46 +0100] "GET /_matrix/client/versions HTTP/1.1" 200 1237 "-" "Element (Riot.im)/1.0.7 (Linux; U; Android 10; Aquaris X Pro Build/QQ3A.200805.00; Flavour GooglePlay; MatrixAndroidSDK_X 0.0.1)"

79.72.35.240 - - [28/Sep/2020:18:43:46 +0100] "GET /_matrix/client/r0/sync?filter=1&timeout=0&since=s8533_21152_0_506_492_1_245_291_1 HTTP/1.1" 200 1290 "-" "Element (Riot.im)/1.0.7 (Linux; U; Android 10; Aquaris X Pro Build/QQ3A.200805.00; Flavour GooglePlay; MatrixAndroidSDK_X 0.0.1)"

79.72.35.240 - - [28/Sep/2020:18:43:46 +0100] "GET /_matrix/client/r0/sync?filter=1&timeout=30000&since=s8533_21152_0_506_492_1_245_291_1 HTTP/1.1" 200 762 "-" "Element (Riot.im)/1.0.7 (Linux; U; Android 10; Aquaris X Pro Build/QQ3A.200805.00; Flavour GooglePlay; MatrixAndroidSDK_X 0.0.1)"

79.72.35.240 - - [28/Sep/2020:18:43:39 +0100] "GET /_matrix/client/r0/sync?filter=1&timeout=30000&since=s8533_21152_0_506_492_1_245_291_1 HTTP/1.1" 200 784 "-" "Element (Riot.im)/1.0.7 (Linux; U; Android 10; Aquaris X Pro Build/QQ3A.200805.00; Flavour GooglePlay; MatrixAndroidSDK_X 0.0.1)"

79.72.35.240 - - [28/Sep/2020:18:44:13 +0100] "POST /_matrix/client/r0/rooms/!test%2Fv6:*******.net/leave HTTP/1.1" 404 1039 "-" "Element (Riot.im)/1.0.7 (Linux; U; Android 10; Aquaris X Pro Build/QQ3A.200805.00; Flavour GooglePlay; MatrixAndroidSDK_X 0.0.1)"

79.72.35.240 - - [28/Sep/2020:18:44:10 +0100] "GET /_matrix/client/r0/sync?filter=1&timeout=30000&since=s8533_21152_0_506_492_1_245_291_1 HTTP/1.1" 200 837 "-" "Element (Riot.im)/1.0.7 (Linux; U; Android 10; Aquaris X Pro Build/QQ3A.200805.00; Flavour GooglePlay; MatrixAndroidSDK_X 0.0.1)"

79.72.35.240 - - [28/Sep/2020:18:44:59 +0100] "POST /_matrix/client/r0/rooms/!test%2Fv6:*******.net/leave HTTP/1.1" 404 1039 "-" "Element (Riot.im)/1.0.7 (Linux; U; Android 10; Aquaris X Pro Build/QQ3A.200805.00; Flavour GooglePlay; MatrixAndroidSDK_X 0.0.1)"

79.72.35.240 - - [28/Sep/2020:18:44:40 +0100] "GET /_matrix/client/r0/sync?filter=1&timeout=30000&since=s8533_21153_0_506_492_1_245_291_1 HTTP/1.1" 200 762 "-" "Element (Riot.im)/1.0.7 (Linux; U; Android 10; Aquaris X Pro Build/QQ3A.200805.00; Flavour GooglePlay; MatrixAndroidSDK_X 0.0.1)"

79.72.35.240 - - [28/Sep/2020:18:45:10 +0100] "GET /_matrix/client/r0/sync?filter=1&timeout=30000&since=s8533_21153_0_506_492_1_245_291_1 HTTP/1.1" 200 837 "-" "Element (Riot.im)/1.0.7 (Linux; U; Android 10; Aquaris X Pro Build/QQ3A.200805.00; Flavour GooglePlay; MatrixAndroidSDK_X 0.0.1)"

167.172.194.42 - - [28/Sep/2020:18:46:09 +0100] "GET /.well-known/matrix/server HTTP/1.1" 404 4130 "-" "Synapse/1.19.1"`

[anoadragon453]

Right, so I did some testing against your server and it seems that you have a slash (%2F) in your room ID. If I remove that, the request goes straight through to Synapse. But with that in there, an Apache error page is returned.

Is there a slash (encoded or not) in the room ID? If so, it doesn't look like Apache handles this too well...

I **d out the remote homeserver (not sure whether that's necessary or not).

Only if you want to hide that that's your domain name. There's not anything anyone could do with that other than query your homeserver without a valid access token. I don't need to see it though.

[mpatey]

The room (I cannot leave) is #test:maunium.net. When I pull up the room properties it lists #test/v6:maunium.net under "Other published addresses", which is the one we see in the apache logs above. Is there some way I can get the client to use the main address rather than the alternative address which has a slash in it?

[anoadragon453]

I see. While there is an alias with a /, the actual room ID contains a / as well, and rooms can only be left using the room ID.

Do you have AllowEncodedSlashes NoDecode in your apache config as specified on https://github.com/matrix-org/synapse/blob/develop/docs/reverse_proxy.md?

[mpatey]

I checked the config, and it was missing AllowEncodedSlashes NoDecode
I have added that line so now I have the following apache config for matrix-synapse:

root@freedombox:~# cat /etc/apache2/conf-enabled/matrix-synapse-plinth.conf
AllowEncodedSlashes NoDecode
ProxyPass /_matrix http://localhost:8008/_matrix nocanon
ProxyPassReverse /_matrix http://localhost:8008/_matrix

After that I did systemctl restart apache2 but I am still getting 404s in the apache log. What's more I have just realised that I cannot send any messages in this room either (I see similar 404 messages).

By the way, what is the room ID? I thought that !test%2Fv6:maunium.net was the (encoded) room alias

[anoadragon453]

After that I did systemctl restart apache2 but I am still getting 404s in the apache log.

NoDecode was added in Apache 2.3.12 according to the docs. It sounds like Apache is still defaulting to the default behaviour of Off, which the docs state "With the default value, Off, such URLs are refused with a 404 (Not found) error."

If you do have an appropriately up-to-date Apache version, for troubleshooting purposes I would try changing NoDecode to On and see whether that changes the behaviour you experience. With that, URLs will be passed through to Synapse at least (however Synapse will return an "Unrecognised Request" error as it wouldn't be able to match the request to a known path pattern).

What's more I have just realised that I cannot send any messages in this room either (I see similar 404 messages).

Yeah, this would make sense as the endpoint for sending messages includes the room ID: PUT /_matrix/client/r0/rooms/{roomId}/send/{eventType}/{txnId}.

By the way, what is the room ID? I thought that !test%2Fv6:maunium.net was the (encoded) room alias

Room aliases and IDs can be differentiated by the leading character. Aliases start with # and look like #something:example.com, whereas a room ID starts with a ! and look like !something:example.com. A room can have multiple aliases, across multiple homeservers, all pointing to the same room. But a room only has one room ID, and it should be unique to that room.

[richvdh]

I'm inclined to close this. It's a combination of @tulir experimenting with specially-crafted room ids (which is unsupported, for this sort of reason) and Apache getting upset about the URL. The bug, if any, is in the matrix spec which fails to declare which characters are valid in a room id (https://github.com/matrix-org/matrix-doc/issues/667).

[mpatey]

If you do have an appropriately up-to-date Apache version, for troubleshooting purposes I would try changing NoDecode to On and see whether that changes the behaviour you experience.

I am running apache 2.4.38, so in theory it should work, but I get the same 404s whether I have AllowEncodedSlashes NoDecode or AllowEncodedSlashes On

I thought that maybe it was because the directive was in the server config section (/etc/apache2/conf-enabled) and somehow wasn't filtering down to the virtual host definition, but adding the directive to the vhost block also doesn't work.

If you want to close this issue then that's fine by me. Thanks for your help in finding out where the problem lies. I will maybe try to see if I can get some help with troubleshooting my apache configuration over at debian/freedombox because it is their out-of-the-box configuration I am using.

It's a combination of @tulir experimenting with specially-crafted room ids (which is unsupported, for this sort of reason)

Out of interest, what would be the use of these specially-crafted room ids?

[richvdh]

Out of interest, what would be the use of these specially-crafted room ids?

"Let's see what breaks".

It may be unrelated, but it sounds like Apache has bugs in this area: https://bz.apache.org/bugzilla/show_bug.cgi?id=35256.