Memory leak

[adhikjoshi]

Everything worked smoothly for weeks and it keeps piling up RAM for weeks.

and sish start giving 502 errors for 20% of requests.

once RAM is full Google Cloud or something triggers RAM cleanup.

RAM goes to 1GB, and all 502 errors are gone.

There's a memory leak happening somewhere.

[antoniomika]

Hi @adhikjoshi

Can you enable debug logging and provide them to me? We run sish in production and can't replicate this issue. I have instances of sish that have been running for years with no restarts and memory is minimal.

My best guess is all of the idle timeouts and deadline protections may be disabled, this would result in connections being kept around and the memory staying large (this also happens if botnets with slow ssh connections attempt to start a handshake but never complete).

This would make sense with the 502 errors as well if using any of the load balancer modes since the state would have bad entries for certain routes, and they'd fail on connection.

Regardless, the debug logs will give us info on the connections state as well as the number of go routines active. Please also include the flags you are running sish with.

Thanks!