CSI: no early return when feasibility check fails on eligible nodes

[tgross]

Fixes #11778

As a performance optimization in the scheduler, feasibility checks
that apply to an entire class are only checked once for all nodes of
that class. Other feasibility checks are "available" checks because
they rely on more ephemeral characteristics and don't contribute to
the hash for the node class. This currently includes only CSI.

We have a separate fast path for "available" checks when the node has
already been marked eligible on the basis of class. This fast path has
a bug where it returns early rather than continuing the loop. This
causes the entire task group to be rejected.

Fix the bug by not returning early in the fast path and instead jump
to the top of the loop like all the other code paths in this method.
Includes a new test exercising topology at whole-scheduler level and a
fix for an existing test that should've caught this previously.

---

Note that the test here is potentially limited in reliability because of the random iterator, but:

$ go test -v ./scheduler -run TestServiceSched_CSITopology -count=100
...
--- PASS: TestServiceSched_CSITopology (0.00s)
PASS
ok      github.com/hashicorp/nomad/scheduler    0.424s

[shoenig]

1 line bug fix with 100 lines of testing; LGTM!

[ygersie]

💯

[github-actions]

I'm going to lock this pull request because it has been closed for 120 days ⏳. This helps our maintainers find and focus on the active contributions.
If you have found a problem that seems related to this change, please open a new issue and complete the issue template so we can capture all the details necessary to investigate further.