hung subprocess causes the agent to hang as well

[clutchski]

Just encountered an agent that was hanging because of a hanging subprocess. This particular case was caused by calling df -k on a hung NFS mount. In this case, we should continue to collect other metrics and just log a disk metrics error. This is also just one instance of many subprocesses that could hang because of broken dependencies.

• add a generic subprocess wrapper that handles all the things we'd want (mandatory timeouts, not trapping SystemExit or signal interrupts, etc).
• add a timeout on each check as a whole. this would be nice because it would also handle network hangs as well.
• something else?

I'd vote for doing the first no matter what, because I've noticed that in general, the exception handling around subprocesses could use some improvement in general. and perhaps the second as well.

[clofresh]

That's odd, I figured the watchdog would have killed that thread. An NFS hang might be at the kernel level and prevent Python-level timeout kills

[clutchski]

The watchdog killed the whole agent. But it would never report any metrics ever, because it hit the NFS mount every loop and hung.

[remh]

Will be fixed by #1892

[yannmh]

Fixeed with #1892