-
-
Notifications
You must be signed in to change notification settings - Fork 317
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Cannot successfully connect presence on selfhosted when using a reverse proxy to handle SSL. #993
Comments
This is what my "messages" tab shows in the network traffic for this connection: {"topic":"realtime:presence","event":"phx_join","payload":{"config":{"broadcast":{"ack":false,"self":false},"presence":{"key":"d6d2b22b-8472-4088-8e0e-4bc6793e2d94"},"postgres_changes":[]},"access_token":"<token>"},"ref":"1","join_ref":"1"}
{"event":"phx_reply","payload":{"response":{"postgres_changes":[]},"status":"ok"},"ref":"1","topic":"realtime:presence"}
{"topic":"realtime:presence","event":"access_token","payload":{"access_token":"<token>"},"ref":"2","join_ref":"1"}
{"topic":"realtime:presence","event":"presence","payload":{"type":"presence","event":"track","payload":{"state":"online"}},"ref":"3","join_ref":"1"}
{"event":"presence_state","payload":{},"ref":null,"topic":"realtime:presence"}
{"event":"presence_diff","payload":{"joins":{"d6d2b22b-8472-4088-8e0e-4bc6793e2d94":{"metas":[{"phx_ref":"F9Cf3GfuHXxMuhIR","state":"online"}]}},"leaves":{}},"ref":null,"topic":"realtime:presence"}
{"event":"phx_reply","payload":{"response":{},"status":"ok"},"ref":"3","topic":"realtime:presence"}
{"event":"phx_error","payload":{},"ref":"1","topic":"realtime:presence"} This set of messages repeats infinitely when connecting to the load balancer. |
Any chance I can get some help looking through this? I even went through the process of directly doing SSL on the server by passing a custom runtime.exs in and linking my certificates. If I connect to the IP directly on port 4000 it works great, but if I connect to my ssl port 4433 I get the exact same errors even though I am no longer behind a proxy. |
Hi @Destreyf do you see the same error when you try to broadcast a message? I suspect that something might be blocking the socket upgrade from HTTP to WS, namely in the location as we use |
I ended up dropping supabase realtime, I was simply trying to use it to sync presence between users, I wrote my own implementation and was able to deploy it behind the same load balancers with zero issue, using the exact same host/path combination. The websocket was upgrading just fine, and was sending/receiving messages, you can see the messages in my comment here: #993 (comment) During my last attempt I listened to SSL directly on port 4433 using supabase realtime, no proxy, no firewalls, nothing, there was literally zero infrastructure/routing between the supabase realtime server and the clients. |
And were you able to see if there was any Realtime log errors? |
The errors are included in my initial post here... the gen server claims the function/handler does not exist, at which point the websocket receives the phx_error payload.
edit: meant the handler_out/3 reference, not process, it's been a long day. |
🤦 sorry late on my timezone got it thank you for the detailed report and we will open PR to fix it 🙏 |
I get it, it's been a long day for myself as well. My original deployment was running on aarch64 (linux/arm64), but I tested this on an x86/64bit machine as well to rule that out. |
It's really interesting that it's hitting a thank you again for the detailed information |
Here's the NGINX config we use for reverse proxy (I'll discuss SSL offloading in afterwards): server {
listen 80;
listen [::]:80;
server_name supabase.ourdomain.com; # change this
client_max_body_size 20M;
large_client_header_buffers 4 32k;
location / {
proxy_pass http://127.0.0.1:8000; # this is where supabase Kong is running
proxy_buffering off;
proxy_redirect off;
proxy_read_timeout 86400; # necessary to avoid websocket timeout disconnect
proxy_http_version 1.1;
proxy_set_header Host $host;
proxy_set_header Upgrade $http_upgrade;
proxy_set_header Connection "upgrade";
proxy_set_header X-Real-IP $remote_addr;
proxy_set_header X-Forwarded-Host $host;
proxy_set_header X-Forwarded-Port $server_port;
proxy_set_header X-Real-IP $remote_addr;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
proxy_set_header X-Forwarded-Proto $scheme;
proxy_set_header Authorization $http_authorization;
proxy_buffer_size 32k;
proxy_buffers 8 32k;
}
} You can manually add SSL, but honestly, Certbot and Lets Encrypt is the way to automate things in production environments. sudo apt-get install -y certbot python3-certbot-nginx
sudo certbot --nginx --non-interactive --agree-tos -m [email protected] -d yourdomain.com -d supabase.yourdomain.com --expand This alone should get you going (we use this exactly in production now) Edit: A bit more explanation: Certbot uses the NGINX config from above and automatically adds everything needed to offload the SSL encryption. After running the above commands, you can see how Certbot adds/modifies your site-enabled |
@barrownicholas I had that exact config at one point as well, it did not work and threw the exact same error messages. The error itself has nothing to do with the proxy layer either as it happened when connecting directly to the realtime service over https with my custom runtime.exs file. Something to note is that I did not use kong, I was only using reatime by itself. |
I agree with @Destreyf , from the error it really seems that Realtime is receiving some payload that is not expected and fails to pattern match so I need to check the error message and understand what payload could be causing it. |
you are a King, thanks! |
Bug report
Describe the bug
I am unable to successfully run the supabase/realtime server behind an nginx-proxy nor the AWS ALB, I have other websocket applications deployed and working behind both of these environments.
If I connect to the server directly via ip:port it works just fine, however when I connect over https using the load balancer endpoint I get the following logs.
The subscribe call emits a
SUBSCRIBED
thenCHANNEL_ERROR
state on thesubscribe
handler.This is effectively the boilerplate demo found in the realtime-js repo: https://github.com/supabase/realtime-js?tab=readme-ov-file#presence
My nginx setup is using nginx-proxy-manager for testing, ideally I would use the ALB in aws, here's my config file for that
I duplicated my "realtime" config to "52" so I can test connecting via IP, which works but obviously isn't through the reverse proxy.
System information
The text was updated successfully, but these errors were encountered: