chore: make load tests more resilient to slow metrics-server startup in GH CI

[btlghrants]

Description

Adds some add'l try/catches+output to allow us to see what's actually blowing up in CI.

Related Issue

Fixes #1622

Type of change

• Bug fix (non-breaking change which fixes an issue)
• New feature (non-breaking change which adds functionality)
• Other (security config, docs update, etc)

Checklist before merging

• Unit, Journey, E2E Tests, docs, adr added or updated as needed
• Contributor Guide Steps followed

[codecov]

Codecov Report

All modified and coverable lines are covered by tests ✅

Project coverage is 81.40%. Comparing base (36b87fe) to head (f3aafb8).
Report is 1 commits behind head on main.

Additional details and impacted files

@@           Coverage Diff           @@
##             main    #1634   +/-   ##
=======================================
  Coverage   81.40%   81.40%           
=======================================
  Files          43       43           
  Lines        1979     1979           
  Branches      438      439    +1     
=======================================
  Hits         1611     1611           
+ Misses        366      340   -26     
- Partials        2       28   +26     

see 7 files with indirect coverage changes

[btlghrants]

Was able to repro & actually see what's been throwing errors:

Suspect this is being caused by the call to kubectl top happening before the kube metrics-server is ready for it:

[btlghrants]

Seems like we're just trying to call for metrics before the metrics-server is fully ready..? https://github.com/kubernetes/kubernetes/blob/master/staging/src/k8s.io/kubectl/pkg/cmd/top/top_pod.go#L207

So... I'm thinking that adding a wait toward the end of the npx ts-node hack/load.cli.ts deploy [...] is probably the way to go here: the definition for a module being "fully deployed" should include having it be available for metrics pulls, IMO.

[btlghrants]

Yeah, now that we have some logging output it's easy to see how large a variance there is in the amount of time it takes for GH to get the metric-server ready to serve controller metrics:

I'll run it the job a few more times to make sure this really solved, but outlook is good.

[btlghrants]

Yeah, nothin's bombin'-out anymore so I think I'm going to call this one good & send it for review.

[cmwylie19]

lgtm