Real time html behind reverse proxy returns HTTP status 400

[elya5]

I have issues setting up the real time view behind a reverse proxy. There are many other issues here about the same topic but I still couldn't figure it out.

Setup: I have a server with Caddy as a reverse proxy and I would like to set up goaccess for it. This server sits behind another server with reverse proxy handling SSL etc. which is available to the public. The outer server forwards the request and rewrites the path (in case that's relevant).
So a user visits https://example.domain/service/goaccess.html and my server only sees http://hostname/goaccess.html.

Now I tried to run goaccess the following way:

goaccess /logs/caddy.log --log-format=CADDY -o /var/www/service/goaccess.html --ws-url=wss://example.domain:443/service/ws --port 7890 --real-time-html

with the caddy directive

handle /ws {
    reverse_proxy localhost:7890
}

When I open the site in the browser, I can see the not-updated statistics and the site correctly tries to connect to wss://example.domain/service/ws, however HTTP status code 400 is returned. Running another service (like python -m http.server 7890), I can see that the request arrives. Apparently goaccess refuses to answer it. Does anyone know what's the issue here?

[0bi-w6n-K3nobi]

Hi @elya5

Humm... It is not so simple like as you think...
The protocol behind URI /ws is WebSocket and not simply HTTP/HTTPS protocol.

Try to search, at internet, about Caddy server with WebSocket.
It is need some steps more for work with it.

I hope that helped you.

[allinurl]

@elya5 Have you looked at this guide for apache? it may help get it going with caddy perhaps. Let us know.

[elya5]

Is there any option for more detailed log about? I already tried --invalid-requests but the requests don't get logged there.

[allinurl]

What's the output from your browser's console? Also, did you look at that guide I posted before? did it help?

[elya5]

Yes, I looked at the guide and the configuration but still couldn't figure it out.

I start goaccess with the following config: goaccess /storage/caddy.log --log-format=CADDY --real-time-html --port 7890 --ws-url "example.domain:443/service/ws" -o /var/www/service/stats2.html.

The browser is sending the following request:

GET /service/ws HTTP/1.1
Host: example.domain
User-Agent: Mozilla/5.0 (X11; Fedora; Linux x86_64; rv:96.0) Gecko/20100101 Firefox/96.0
Accept: */*
Accept-Language: de-DE,de;q=0.8,en-US;q=0.5,en;q=0.3
Accept-Encoding: gzip, deflate, br
Sec-WebSocket-Version: 13
Origin: https://example.domain
Sec-WebSocket-Extensions: permessage-deflate
Sec-WebSocket-Key: wrCezY98yE0pJ4QaKJnG0A==
DNT: 1
Authorization: Basic abcdefg
Connection: keep-alive, Upgrade
Sec-Fetch-Dest: websocket
Sec-Fetch-Mode: websocket
Sec-Fetch-Site: same-origin
Pragma: no-cache
Cache-Control: no-cache
Upgrade: websocket

And receives the following response:

HTTP/1.1 400 Bad Request
Date: Tue, 01 Feb 2022 08:06:01 GMT
Server: Caddy
Via: 1.1 example.domain
Vary: Accept-Encoding,User-Agent
Content-Encoding: gzip
Connection: close
Transfer-Encoding: chunked

If I stop goaccess and run netcat on the same port, I receive the following request:

# nc -l -p 7890 -s 0.0.0.0
GET /service/ws HTTP/1.1
Host: service21.example.domain
User-Agent: Mozilla/5.0 (X11; Fedora; Linux x86_64; rv:96.0) Gecko/20100101 Firefox/96.0
Accept: */*
Accept-Encoding: gzip, deflate, br
Accept-Language: de-DE,de;q=0.8,en-US;q=0.5,en;q=0.3
Authorization: Basic abcdefg
Cache-Control: no-cache
Dnt: 1
Origin: https://example.domain
Pragma: no-cache
Sec-Fetch-Dest: websocket
Sec-Fetch-Mode: websocket
Sec-Fetch-Site: same-origin
Sec-Websocket-Extensions: permessage-deflate
Sec-Websocket-Key: 6zH1pDuqEfs9vQeTQrFAvw==
Sec-Websocket-Version: 13
Via: 1.1 example.domain
X-Forwarded-For: 95.90.200.187, 194.95.114.13
X-Forwarded-Host: example.domain
X-Forwarded-Proto: https
X-Forwarded-Server: example.domain
X_forwarded_port: 443
X_https: on
X_ssl_protocol: TLSv1.2

[demifiend9]

@elya5 not sure if this is helpful or relevant anymore, but I'm using caddy with reverse proxy and websocket works

goaccess config

real-time-html true
addr 127.0.0.1
port 15564
ws-url stats.example.com:443
log-format CADDY
log-file D:\caddy\logs\example.com.log.json
output D:\caddy\sites\stats.example.com\index.html

caddy config

stats.example.com {
	@localonly not remote_ip 192.168.15.0/24 10.0.15.0/24
	respond @localonly bruh 403
	@goaccessws {
		header Connection *Upgrade*
		header Upgrade websocket
		remote_ip 192.168.15.0/24 10.0.15.0/24
	}
	reverse_proxy @goaccessws 127.0.0.1:15564
	file_server / {
		root sites\stats.catto.win
	}
}

Try removing wss:// part from --ws-url option. Another problem I see in your config is that goaccess is expecting to serve websocket on example.domain as specified by your --ws-url but you're reverse proxying on localhost. Try adding --addr=localhost

[allinurl]

@demifiend9 Thanks a lot for sharing that. @elya5 did that help?

[elya5]

Thanks @demifiend9 for the info. I probably missed the connection upgrade, however even with the changes it still does not work.

Here is the caddy config part:

@goaccessws {
	header Connection *Upgrade*
	header Upgrade websocket
}
handle /ws {
    reverse_proxy @goaccessws localhost:7890 
}

The goaccess call: goaccess /permanent/caddy.log --log-format=CADDY -o /var/www/service/goaccess.html --ws-url=example.domain:443/service/ws --addr=localhost --port 7890 --real-time-html

The browser shows the following request:

GET /service/ws HTTP/1.1
Host: example.domain
User-Agent: Mozilla/5.0 (X11; Fedora; Linux x86_64; rv:97.0) Gecko/20100101 Firefox/97.0
Accept: */*
Accept-Language: de-DE,de;q=0.8,en-US;q=0.5,en;q=0.3
Accept-Encoding: gzip, deflate, br
Sec-WebSocket-Version: 13
Origin: https://example.domain
Sec-WebSocket-Extensions: permessage-deflate
Sec-WebSocket-Key: LxocQSF6HnuY7JQ3uY826w==
DNT: 1
Authorization: Basic abcdefg
Connection: keep-alive, Upgrade
Sec-Fetch-Dest: websocket
Sec-Fetch-Mode: websocket
Sec-Fetch-Site: same-origin
Pragma: no-cache
Cache-Control: no-cache
Upgrade: websocket

and the following response:

HTTP/1.1 200 OK
Date: Tue, 22 Feb 2022 09:20:22 GMT
Server: Caddy
Content-Length: 0
Via: 1.1 example.domain
Vary: User-Agent
Keep-Alive: timeout=5, max=100
Connection: Keep-Alive

It seems that maybe Caddy is directly responding to request without forwarding it to goaccess.

[0bi-w6n-K3nobi]

Hi @elya5

Maybe your URL into caddy config for WS be incorrect.

I think you must declare /service/ws instead of /ws... or change it at GOACCESS command-line.

humm... Another point is if you use port 443 then need use WSS, and set correct SSL Certificate
with --ssl-cert and --ssl-key options.
See more at man page here at SERVER OPTIONS section.

[demifiend9]

Thanks @demifiend9 for the info. I probably missed the connection upgrade, however even with the changes it still does not work.

Here is the caddy config part:

@goaccessws {
	header Connection *Upgrade*
	header Upgrade websocket
}
handle /ws {
    reverse_proxy @goaccessws localhost:7890 
}

@elya5 You don't need to use /ws path for hosting the webscoket as caddy will match the websocket headers to correctly reverse proxy it. remove /ws from both configs

Try this.

goaccess /logs/caddy.log --log-format=CADDY -o /var/www/service/goaccess.html --ws-url=example.domain:443/service --addr 127.0.0.1 --port 7890 --real-time-html

@goaccessws {
	header Connection *Upgrade*
	header Upgrade websocket
}
reverse_proxy @goaccessws 127.0.0.1:7890 

If it still doesn't work then it's probably a problem with /service path of your proxy server above caddy. Reverse proxies are known to not play well with paths, thats why it's preferred to use a different subdomain to reverse proxy to, instead of url path like /service.

You can test if it indeed is a problem with /service url by trying to caddy reverse proxy into any other application (for example qbittorrent webui) and see if they work. You can also try to remove /service url completely from your proxy and host goacces directly under / just to test if the websocket works. Another way to do this is using uri strip_prefix.