Fix metrics controller races

[tzneal]

1. Issue, if available:
N/A

2. Description of changes:
add some race testing and fix two identified issues

3. How was this change tested?
ginkgo -r --race

4. Does this change impact docs?

• Yes, PR includes docs updates
• Yes, issue opened: link to issue
• No

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.

[netlify]

✔️ Deploy Preview for karpenter-docs-prod canceled.

🔨 Explore the source changes: fa560ee

🔍 Inspect the deploy log: https://app.netlify.com/sites/karpenter-docs-prod/deploys/621e8fc178b08100081ed257

[ellistarn]

I'm a bit wary of ExpectRaceFree, though I'll admit it's pretty cool.

• I'd prefer to catch this sort of thing in e2e tests, where the raceable scope is much larger. I recognize that we won't be using -race in this environment, though
• I worry that this might give us a false sense of security that our tests won't race
• In general, testing that something doesn't happen I really hard, since there are an infinite number of things that won't happen. I could see an argument for specific cases like paralleism/racing, but I'd prefer to limit our package testing to behavior checking to catch regressions.
• I'm a bit wary of the time added to CI tests (how fast is this?)

[tzneal]

I'm a bit wary of ExpectRaceFree, though I'll admit it's pretty cool.

• I'd prefer to catch this sort of thing in e2e tests, where the raceable scope is much larger. I recognize that we won't be using -race in this environment, though

You can build with -race, though it will run much slower. That's probably the best solution as a -race e2e test should negate the need for this entirely.

• I worry that this might give us a false sense of security that our tests won't race

• In general, testing that something doesn't happen I really hard, since there are an infinite number of things that won't happen. I could see an argument for specific cases like paralleism/racing, but I'd prefer to limit our package testing to behavior checking to catch regressions.

On the other hand, the test does expose two bugs :)

• I'm a bit wary of the time added to CI tests (how fast is this?)

The 50 objects * 5 runs was to hopefully trip an issue like a concurrent map write during normal unit tests. You can just create two objects and run them once to trigger detection with the race detector, so time should be negligible in that case.

Possible solution is to just fix the two bugs and create an issue to ensure that at least some e2e test runs on a -race build?

[ellistarn]

Possible solution is to just fix the two bugs and create an issue to ensure that at least some e2e test runs on a -race build?

Perfect.