cmd: Log goroutines (stack traces) on SIGUSR1

[mitjat]

Task

Produce core dump (goroutines stack trace) on demand

Perhaps a custom handler for SIGUSR1 that dumps the core into a file and/or logs it.

(For services (= "the API"), there's a fairly standard way in Go to add an endpoint, but I'm not sure what the best way would be to expose the endpoint only internally. Might not be worth the additional hassle; we've never needed a core dump there.)

Motivation:

Currently, the only way to obtain the stacks of goroutines is to send a SIGQUIT, which also terminates the program. And in a way that does not safely close the KVStore 💀, at that. When analyzers get stuck, the list of goroutines can be very helpful for diagnosing the live system.

This PR

This PR maxes the nexus binary respond to the SIGUSR1 os signal by logging the stack traces of all goroutines.

A good way to read the stack traces is to pass the (last minute of) logs through on of:

| jq -r  '.goroutines_all | select(.)'
| jq -r  '.goroutines_block | select(.)'
| jq -r  '.goroutines_mutex | select(.)'

Tested locally.

[pro-wh]

do we need to tell the go runtime that we don't have to wait for this goroutine to exit before exiting?

[mitjat]

do we need to tell the go runtime that we don't have to wait for this goroutine to exit before exiting?

I don't think so. It hasn't come up in my testing. I believe Go doesn't have an implicit all_goroutines.Wait() appended to the end of main(). See also this mini demo; it prints "Program exited" immediately.

[pro-wh]

ok