roachtest: nightly suite runner is extremely overloaded, causes test failures

[tbg]

For a long time, we've had issues with tests in the roachtest nightly suite failing in a mysterious way that suggested that a time.Sleep(time.Second) was sometimes taking much (>>20x) longer.

I finally got to the bottom of it and the answer appears to be that the roachtest runner machine is severely overloaded: CPU utilization is at 100%, the machine is almost out of memory, and heap GC runs take north of a minute.

It's surprising that anything works at all under these circumstances, but I'm sure it causes all kinds of spurious failures across many tests that we likely shrug off as "infra flakes".

We need to find out why this machine is so overloaded, fix that, and set up monitoring for the health of the roachtest machine.

#138864 has the in-depth analysis. The execution trace referred in that analysis is in this slack thread (in case the artifacts are already gone).

Jira issue: CRDB-46415

[blathers-crl]

cc @cockroachdb/test-eng

[RaduBerinde]

Once we address this, we could set up a background goroutine in roachtest which keeps scheduling a timer and panics if it's awakened more than e.g. 10s than expected. This would at least save us from having to debug spurious test failures.