Accepted connection spam in sish logs when trying to use a TCP tunnel from within a docker container

[richturner]

Hi,

Thanks for this great project!

I have an interesting situation where I am trying to create a TCP tunnel to SISH from within a docker container, the tunnel establishes OK but when I make a request that uses it the response stops half way through and in the sish logs I just see new socket descriptors being constantly created until I close the tunnel:

2024/06/27 - 15:31:15 | Accepted connection from xxx.xxx.35.59:55619 -> xxx.xxx.0.6:9001
2024/06/27 - 15:31:15 | Accepted connection from xxx.xxx.35.59:55620 -> xxx.xxx.0.6:9001
2024/06/27 - 15:31:15 | Accepted connection from xxx.xxx.35.59:55621 -> xxx.xxx.0.6:9001
2024/06/27 - 15:31:15 | Accepted connection from xxx.xxx.35.59:55622 -> xxx.xxx.0.6:9001
2024/06/27 - 15:31:15 | Accepted connection from xxx.xxx.35.59:55623 -> xxx.xxx.0.6:9001
2024/06/27 - 15:31:15 | Accepted connection from xxx.xxx.35.59:55624 -> xxx.xxx.0.6:9001
2024/06/27 - 15:31:15 | Accepted connection from xxx.xxx.35.59:55625 -> xxx.xxx.0.6:9001
2024/06/27 - 15:31:16 | Accepted connection from xxx.xxx.35.59:55626 -> xxx.xxx.0.6:9001
2024/06/27 - 15:31:16 | Accepted connection from xxx.xxx.35.59:55627 -> xxx.xxx.0.6:9001
2024/06/27 - 15:31:16 | Accepted connection from xxx.xxx.35.59:55628 -> xxx.xxx.0.6:9001
2024/06/27 - 15:31:16 | Accepted connection from xxx.xxx.35.59:55630 -> xxx.xxx.0.6:9001
2024/06/27 - 15:31:16 | Accepted connection from xxx.xxx.35.59:55631 -> xxx.xxx.0.6:9001
2024/06/27 - 15:31:16 | Accepted connection from xxx.xxx.35.59:55632 -> xxx.xxx.0.6:9001
2024/06/27 - 15:31:16 | Accepted connection from xxx.xxx.35.59:55633 -> xxx.xxx.0.6:9001
2024/06/27 - 15:31:16 | Accepted connection from xxx.xxx.35.59:55634 -> xxx.xxx.0.6:9001
2024/06/27 - 15:31:16 | Accepted connection from xxx.xxx.35.59:55635 -> xxx.xxx.0.6:9001
2024/06/27 - 15:31:16 | Accepted connection from xxx.xxx.35.59:55636 -> xxx.xxx.0.6:9001
2024/06/27 - 15:31:16 | Accepted connection from xxx.xxx.35.59:55637 -> xxx.xxx.0.6:9001
2024/06/27 - 15:31:16 | Accepted connection from xxx.xxx.35.59:55638 -> xxx.xxx.0.6:9001
2024/06/27 - 15:31:17 | Accepted connection from xxx.xxx.35.59:55639 -> xxx.xxx.0.6:9001
2024/06/27 - 15:31:17 | Accepted connection from xxx.xxx.35.59:55640 -> xxx.xxx.0.6:9001
2024/06/27 - 15:31:17 | Accepted connection from xxx.xxx.35.59:55641 -> xxx.xxx.0.6:9001

The SSH command I am using to create the tunnel within a docker container is as follows:

ssh -vvv -i /my/cert -p 2222 -R 9001:192.168.5.1:80 my.server.com

If I use the exact same command outside of docker then it seems to work fine.

When running the command within a docker container I see this constantly repeating in the output:

debug1: channel 23: new [localhost]
debug1: confirm forwarded-tcpip
debug3: channel 23: waiting for connection
debug1: channel 23: connected to 192.168.5.1 port 80
debug3: send packet: type 91
debug3: receive packet: type 90
debug1: client_input_channel_open: ctype forwarded-tcpip rchan 24 win 2097152 max 32768
debug1: client_request_forwarded_tcpip: listen localhost port 9001, originator localhost port 9001
debug2: fd 30 setting O_NONBLOCK
debug2: fd 30 setting TCP_NODELAY
debug1: connect_next: host 192.168.5.1 ([192.168.5.1]:80) in progress, fd=30
debug3: fd 30 is O_NONBLOCK
debug3: fd 30 is O_NONBLOCK
debug1: channel 24: new [localhost]
debug1: confirm forwarded-tcpip
debug3: channel 24: waiting for connection
debug1: channel 24: connected to 192.168.5.1 port 80
debug3: send packet: type 91
debug3: receive packet: type 90
debug1: client_input_channel_open: ctype forwarded-tcpip rchan 25 win 2097152 max 32768
debug1: client_request_forwarded_tcpip: listen localhost port 9001, originator localhost port 9001
debug2: fd 31 setting O_NONBLOCK
debug2: fd 31 setting TCP_NODELAY
debug1: connect_next: host 192.168.5.1 ([192.168.5.1]:80) in progress, fd=31
debug3: fd 31 is O_NONBLOCK
debug3: fd 31 is O_NONBLOCK

I'm running out of ideas on this one so hoping someone may have some insights that could help.

[antoniomika]

Hi @richturner,

Thanks for the kind words!

This is an interesting one, mind sharing what TCP app you’re testing? I’m wondering if there’s something we should be doing on the tunnel side that clients for that app expect. The other follow-up was just ensuring the docker container can access 192.168.5.1:80 fine, but it sounds like you’ve already verified that.

Best,

[richturner]

Thanks for the reply.

Indeed the IP address is reachable from within the docker container and a curl command from within the container works fine.

The service I was trying to access is actually a router HTTP UI (dd-wrt) but I tried a couple of different router web UIs on different LANs and the curl command always hangs at the same point here:

$ curl -v http://test3.openremote.app:9001
*   Trying xxx.xxx.8.193:9001...
* Connected to my.server.com (xxx.xxx.8.193) port 9001 (#0)
> GET / HTTP/1.1
> Host: my.server.com:9001
> User-Agent: curl/7.87.0
> Accept: */*
>

After your comment I have now tried with an actual raw TCP server and that seems to work fine so it is something to do with the combination of curl and/or the HTTP server(s), but given I have tried multiple HTTP servers with the same result and also the fact that the curl works fine when tunnel is established outside of a docker container it suggests there is something subtle happening here with the interaction of docker. Worth noting I tried running the docker container with host network mode (to eliminate docker networking and the same error situation occurs).

It was accidental using a TCP tunnel to access a HTTP server as it was just quicker to use for testing as HTTP servers were already running to test against.

It is clear that this failure mode is pretty bad as the rate of socket descriptor creation is very quick and doesn't stop until I terminate the tunnel, terminating the curl request does not stop it.

Hopefully this helps shed some light on what might be causing this situation, I guess for now we should avoid accessing HTTP servers over TCP tunnels through a docker container.

Thanks again and let me know if there's anything I can do.