node drain default behavior for -ignore-system

[tgross]

In #8606 (comment) @jippi raised the question of whether nomad node drain should have the -ignore-system flag set by default.

Currently the default behavior is that service and batch jobs are drained, and then system jobs are drained. However, internal allocation runner post-run hooks (ex. deregistering from Consul, cleaning up disk) don't block draining the system jobs, so they may run concurrently with the system jobs being drained. With the -ignore-system flag, the system jobs are never drained, which makes them available during shutdown.

The -ignore-system flag appears to be strictly more useful than not having it set. Some example scenarios where you'd want to have a system job that runs until all other workloads have completed:

• log shippers
• monitoring agents
• ingress proxies (ex. Nginx or HAProxy in front of web services)
• CSI node plugins

Changing the default behavior would break backwards compatibility, so we want to solicit feedback from the community about whether this change would be disruptive.

[josh-m-sharpe]

💯 💯 💯