Nomad UI Bug: CPU Total Displays as 0 When Task Resources Use cores

[ynl]

Nomad version

1.9.4-dev

Operating system and Environment details

Linux

Issue

When a task's resources block specifies CPU using cores, the allocation monitoring UI incorrectly displays "0 Total" CPU under the Resources tab.

Reproduction steps

Expected Result

Actual Result

Job file (if appropriate)

job "cpu-test" {
  datacenters = ["dc1"]
  type = "service"

  group "test" {
    count = 1

    task "cpu-burn" {
      driver = "docker"

      config {
        image = "alpine:latest"
        command = "/bin/sh"
        args = [
          "-c",
          "for i in $(seq 1 6); do dd if=/dev/zero of=/dev/null & done; sleep 3600"
        ]
      }

      resources {
        cores = 6   
        memory = 256
      }
    }
  }
}

Nomad Server logs (if appropriate)

Nomad Client logs (if appropriate)

[Hendrik-vandenBoogaard-Intergas]

We see the same, but we don't specify cores. We do however specify cpu so if a job goes over the limit we get a prometheus based alert that the job is using more than its allocated CPU.

Currently, those jobs hover around 0% but before (Nomad 1.9.3) their value worked ok.

Anonymized job file:

job "mqtt-processor" {
    type = "service"
    region = "global"
    datacenters = ["spanned-dc-production"]
    meta { run_uuid = "${uuidv4()}" 

    group "mqtt-processor" {

        count = 1

        network { port "mqtt-metrics-port" { to = 9095 } }
        task "mqtt-processor" {
            driver = "docker"
            config {
                image = xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
                force_pull = true
                ports = ["mqtt-metrics-port"]
                dns_servers = ["${attr.unique.network.ip-address}"]
            }

            service {
                name = "mqtt-processor"
                port = "mqtt-metrics-port"
                tags = [ "prometheus_target" ]
                check {
                    type     = "http"
                    path     = "/"
                    interval = "10s"
                    timeout  = "2s"
                }
            }

            env {
                xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
            }

            resources {
                cpu    = 1500
                memory = 128
                memory_max = 512
            }
        }
    }
}

Grafana output of job CPU consumption on 1.9.3 vs 1.9.4:

docker stats also shows that the CPU is a lot more busy:

CONTAINER ID   NAME                                                          CPU %     MEM USAGE / LIMIT   MEM %     NET I/O           BLOCK I/O      PIDS
a37a9e6dd58b   mqtt-processor-b3910bbe-b277-ccce-7d1a-b156ddd04cd2           30.49%    79.34MiB / 512MiB   15.50%    3.03GB / 11.4GB   0B / 0B        14

[philrenaud]

@ynl We have a DefaultResources() block that gets returned to the UI in the event that no resources are defined: https://github.com/hashicorp/nomad/blob/main/api/resources.go#L64-L70

But it looks like the CPU default isn't included if Cores is defined (one precludes the other).

I am thinking about what to show in the UI in this case where the task doesn't have a reserved amount. I could show the CPU's total (like what the client page shows), but that feels a little disingenuous. Might just show it without a divisor. What do you think?

=================

@Hendrik-vandenBoogaard-Intergas I am not sure your comment shows the same issue, though it does appear to show a different bug. I'm going to raise that internally and see if anyone else has noticed a CPU reporting change from 1.9.x.