Add "spread" behavior for job placement

[chuyskywalker]

As written here Nomad currently employs a 'bin packing' algorithm for job placement. Meaning that, as you run more and more jobs Nomad will always try to "fill up" the first node before placing jobs on other nodes. While most efficient, it has a utilization/risk trade off (covered better over herehere).

Currently Nomad is lacking the functionality to allow operators to choose how to distribute jobs between these two options.

[chuyskywalker]

My current workaround for this is something like:

job "jobname" {
    type = "service"
    datacenters = [ "dc1" ]
    group "groupname" {
        # Attempt to always be running 2 copies, but not on the same host
        count = 2
        constraint {
            distinct_hosts = true
        }
        task "taskname" {
            driver = "docker"
            config {
                image = "docker/image"
            }
            resources {
                memory = 256
            }
        }
    }
}

This emulates the spread behavior, but only a bit. I still end up with my first two nodes bearing all the burden while my third node sits idly by. It also doesn't take into account the fact that, even if I were down to a single node, I'd want 2 copies running -- Nomad would be unable to maintain that. Granted, it's a bit silly since in 1-of-3 surviving situation, Nomad would lose consensus and "halt". However, if you consider desiring 5 containers in a 7 node cluster -- lose 3 machines, keep consensus, but only be able to run 4 of the desired containers due to constraints, that's perhaps a better illustrations of how this workaround begins to fail.

[dadgar]

Hey,

I would like to clarify. Within a job there is an implicit spread policy, but cluster wide behavior is to bin-pack. So in your example, if you do not explicitly need to have them run on distinct hosts you can just forgo that option.

I am going to close this because we do not want to promote a spread behavior as it is suboptimal scheduling behavior for a cluster as a whole and within jobs this already exists.

Thanks,
Alex

[github-actions]

I'm going to lock this issue because it has been closed for 120 days ⏳. This helps our maintainers find and focus on the active issues.
If you have found a problem that seems similar to this, please open a new issue and complete the issue template so we can capture all the details necessary to investigate further.