Return FailedTGAlloc metric instead of no node err #6968
Conversation
| } | ||
|
|
||
| s.failedTGAllocs[missing.TaskGroup.Name] = s.ctx.Metrics() | ||
| continue |
There was a problem hiding this comment.
Alternatively we could just log the error here and then call s.stack.SetNodes(nodes) with zero nodesto allow stack.Select to occur and possibly populate more Metrics
There was a problem hiding this comment.
This way we get a record of which node was missing when we tried to place the alloc, though, right? That seems better than the alternative.
There was a problem hiding this comment.
Previously we'd return early but now we're collecting the metrics info (which is great!) but then continuing on. From the operator's perspective does this mean that we're going to have a partial placement?
There was a problem hiding this comment.
Yes, in the case of a system job running on 2 nodes, marking 1 node ineligible the resulting plan is
→ nomad job plan repro.hcl
+/- Job: "redis"
+/- Task Group: "cache" (2 create/destroy update)
+/- Task: "redis" (forces create/destroy update)
+/- Env[version]: "1" => "2"
Scheduler dry-run:
- WARNING: Failed to place allocations on all nodes.
Task Group "cache" (failed to place 1 allocation):
Job Modify Index: 13
To submit the job with version verification run:
nomad job run -check-index 13 repro.hcl
When running the job with the check-index flag, the job will only be run if the
server side version matches the job modify index returned. If the index has
changed, another user has modified the job and the plan's results are
potentially invalid.
| } | ||
|
|
||
| s.failedTGAllocs[missing.TaskGroup.Name] = s.ctx.Metrics() | ||
| continue |
There was a problem hiding this comment.
This way we get a record of which node was missing when we tried to place the alloc, though, right? That seems better than the alternative.
If an existing system allocation is running and the node its running on is marked as ineligible, subsequent plan/applys return an RPC error instead of a more helpful plan result. This change logs the error, and appends a failedTGAlloc for the placement.
drewbailey
force-pushed
the
b-system-sched-plan-ineligible
branch
from
19f8302 to
abde9f9
Compare
|
I'm going to lock this pull request because it has been closed for 120 days ⏳. This helps our maintainers find and focus on the active contributions. |
If an existing system allocation is running and the node its running on
is marked as ineligible, subsequent plan/applys return an RPC error
instead of a more helpful plan result.
This change logs the error, and appends a failedTGAlloc for the
placement.
fixes #5169