Log client node plan rejection at INFO instead of DEBUG #11191
Conversation
There was a problem hiding this comment.
What action do you expect users to take in response to this log line?
Plan rejection should be an infrequent but unexceptional outcome of Nomad's optimistically concurrent scheduling: when 2 scheduler workers generate plans simultaneously they may have conflicts. The plan applier's job is to process plans one-to-at-a-time and reject conflicting plans (so the second one processed).
When a plan is rejected the evaluation is reprocessed against the new state of the cluster and a new plan is submitted.
Therefore there's really nothing user actionable about plan rejections: they're generated by Nomad and resolved by Nomad. A user has no control over them.
If a plan is being rejected erroneously due to a bug in the scheduler logic, we should fix that. If we wish to increase visibility into these bugs I think ticking a metric every time a plan is rejected would be easier to notice then increasing log verbosity. If the metric spikes, especially in the absence of high scheduler load, then the nomad operator debug or nomad monitor commands can be used to see exactly what is being rejected and why.
Sadly we now know the answer to this: restart the node in the log line. By the time we verified this bug and discovered the workaround I had forgotten about this issue entirely and submitted basically the same thing in #11416. Closing this as dupe. Dave was right! We'll get to the root cause of #9506 ASAP. |
|
I'm going to lock this pull request because it has been closed for 120 days ⏳. This helps our maintainers find and focus on the active contributions. |
When a Nomad client node rejects a plan to place an allocation, such as the case when a port collision has occurred, Nomad currently logs the plan rejection at DEBUG level. To improve visibility and debuggability this PR increases the logging level to INFO.