Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

System jobs don't get re-evaluated when node attributes change #8448

Closed
notnoop opened this issue Jul 16, 2020 · 1 comment · Fixed by #11007
Closed

System jobs don't get re-evaluated when node attributes change #8448

notnoop opened this issue Jul 16, 2020 · 1 comment · Fixed by #11007

Comments

@notnoop
Copy link
Contributor

notnoop commented Jul 16, 2020

If a node is initially ineligible for a system job, the node will not get scheduled an allocation when it becomes eligible for one.

The reason is that the node doesn't get re-evaluated unless the node update updates status in https://github.com/hashicorp/nomad/blob/v0.12.0/nomad/node_endpoint.go#L172-L185 .

Reproduction steps

  1. disable the docker engine on the host
  2. start nomad agent -dev
  3. run nomad job run ./example.nomad - notice that nomad job status example reports no allocations
  4. enable the docker engine, and wait for nomad node status -self to report the driver as healthy
  5. note that nomad job status example doesn't report any allocation on the node

A workaround is to force evaluation of the job or re-submit, and it will be scheduled on the node as expected.

Job file (if appropriate)

job "example" {
  datacenters = ["dc1"]
  type = "system"

  group "cache" {
    task "redis" {
      driver = "docker"

      config {
        image = "redis:3.2"
      }

      resources {
        cpu    = 500
        memory = 256
      }
    }
  }
}
notnoop pushed a commit that referenced this issue Aug 10, 2021
Fix a bug where system jobs may fail to be placed on a node that
initially was not eligible for system job placement.

This changes causes the reschedule to re-evaluate the node if any
attribute used in feasibility checks changes.

Fixes #8448
notnoop pushed a commit that referenced this issue Aug 26, 2021
Fix a bug where system jobs may fail to be placed on a node that
initially was not eligible for system job placement.

This changes causes the reschedule to re-evaluate the node if any
attribute used in feasibility checks changes.

Fixes #8448
@github-actions
Copy link

I'm going to lock this issue because it has been closed for 120 days ⏳. This helps our maintainers find and focus on the active issues.
If you have found a problem that seems similar to this, please open a new issue and complete the issue template so we can capture all the details necessary to investigate further.

@github-actions github-actions bot locked as resolved and limited conversation to collaborators Oct 17, 2022
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Projects
None yet
Development

Successfully merging a pull request may close this issue.

1 participant