When there are blocked evaluations, nomad.nomad.blocked_evals.[cpu,memory] are always 0

[protochron]

Nomad version

Nomad v1.3.0

Operating system and Environment details

Nomad running on Debian on AWS

Issue

I'm trying to use nomad.nomad.blocked_evals.memory as information to pass to the Nomad autoscaler to scale out the cluster, but the metric is always unset.

Ex.

$ curl localhost:4646/v1/metrics?format=prometheus 2>/dev/null | grep "blocked_evals"
# HELP nomad_nomad_blocked_evals_job_cpu nomad_nomad_blocked_evals_job_cpu
# TYPE nomad_nomad_blocked_evals_job_cpu gauge
nomad_nomad_blocked_evals_job_cpu{job="example",namespace="default"} 0
# HELP nomad_nomad_blocked_evals_job_memory nomad_nomad_blocked_evals_job_memory
# TYPE nomad_nomad_blocked_evals_job_memory gauge
nomad_nomad_blocked_evals_job_memory{job="example",namespace="default"} 3000
# HELP nomad_nomad_blocked_evals_total_blocked nomad_nomad_blocked_evals_total_blocked
# TYPE nomad_nomad_blocked_evals_total_blocked gauge
nomad_nomad_blocked_evals_total_blocked 1
# HELP nomad_nomad_blocked_evals_total_escaped nomad_nomad_blocked_evals_total_escaped
# TYPE nomad_nomad_blocked_evals_total_escaped gauge
nomad_nomad_blocked_evals_total_escaped 0
# HELP nomad_nomad_blocked_evals_total_quota_limit nomad_nomad_blocked_evals_total_quota_limit
# TYPE nomad_nomad_blocked_evals_total_quota_limit gauge
nomad_nomad_blocked_evals_total_quota_limit 0

It looks like the per-job stats are being populated, but the per-node stats are not being populated. Is there something I'm missing here, since per-node stats are enabled in the telemetry block and I'd expect these values to be non-zero if there are blocked allocations.

Reproduction steps

Try and deploy a Nomad job with a memory resource request that exceeds what is available in a cluster.

Expected Result

Non-zero metrics

Actual Result

Empty metrics

[schmichael]

Hi Dan, sorry for the frustration. I'm afraid we use the word "blocked" at least 3 different ways in metrics (if you're curious about our attempts to fixup at least one use: #6480).

From the example you posted it does seem like blocked_evals.memory is being set though:

nomad_nomad_blocked_evals_job_memory{job="example",namespace="default"} 3000

Note that this metric will only be emitted by the leader server in Nomad clusters as that is where blocked evaluations are tracked.

That number is intended to be useful for autoscaling, so I think you're looking at the right "blocked." Note that as recently discussed in #13740 that if you're using quotas (enterprise only), then jobs that are blocked because they would exceed their quota limit are not counted in this metric. The idea is that you don't want your autoscaler to increase cluster capacity when it's the quota that's blocking placement!

Since the nomad.nomad.blocked_evals.{cpu,memory} metrics are incremented when jobs cannot be scheduled, there is no node label to attach to them.

Perhaps you were think of the nomad.client.allocations.blocked metric mentioned here: https://www.nomadproject.io/docs/operations/metrics-reference#host-metrics Unfortunately that's another of our misuses of the word "blocked." It really means "waiting on a previous alloc", and I've clarified that here: #13769

I'm going to close this for now since I think you have what you need, but please do not hesitate to reopen it if you have further questions or issues!

[protochron]

@schmichael thanks for the info and the quick response!

I may not have been clear enough in what I'm asking about. While I can get things working with blocked_evals_job_memory, the metric isn't quite as granular as I'd like it to be for cluster autoscaling since the labels only include the namespace, and not the node_class or datacenter.

hashicorp/nomad-autoscaler#584 is effectively what I'm after here -- the ability to clue the autoscaler in to the fact that there are blocked evals, either split out by datacenter or node_class. For now I have a workaround using blocked_evals_job_memory, but at some point that isn't going to work for us.

I tested this by creating a job that could never be scheduled due to available memory in the cluster, and was surprised that nomad.nomad.blocked_evals.{cpu,memory} was never set! From what you said, it sounds like that metric should have been non-zero unless I'm missing some additional subtlety or time delay.

My sample job for reference:

job "example" {
  datacenters = ["workers"]

  group "cache" {
    network {
      port "db" {
        to = 6379
      }
    }

    task "redis" {
      driver = "docker"
      config {
        image = "redis:3.2"

        ports = ["db"]
      }

      resources {
        cpu    = 500
        
        # This exceeds the free memory in the cluster and sets the metrics I outlined in the issue.
        memory = 5000
      }
    }
  }
}

[schmichael]

Ah! I confused nomad_nomad_blocked_evals_job_memory (note that job) and the node/dc oriented metrics you're referring to (nomad.nomad.blocked_evals.{cpu,memory})! Sorry for the confusion and thanks for sticking with this. It does indeed seem like a bug to me as my simple attempt at repro didn't produce any non-job blocked-evals metrics:

Assuming the datacenter exists and that there is not enough memory available to satisfy the request I get:

~$ curl -s localhost:4646/v1/metrics?format=prometheus | rg -v '^#' | rg blocked_eval
nomad_nomad_blocked_evals_job_cpu{host="dean",job="example",namespace="default"} 0
nomad_nomad_blocked_evals_job_memory{host="dean",job="example",namespace="default"} 50000 <--
nomad_nomad_blocked_evals_total_blocked{host="dean"} 1                                    <--
nomad_nomad_blocked_evals_total_escaped{host="dean"} 0
nomad_nomad_blocked_evals_total_quota_limit{host="dean"} 0

Note only the _job_ variant metrics!

However looking at the evaluation I see it evaluated a node:

d$ nomad eval status -json 8057d3dd
{
    "ClassEligibility": {
        "v1:8536113574940290969": true
    },
    "EscapedComputedClass": false,
    "FailedTGAllocs": {
        "cache": {
            "AllocationTime": 184992,
            "DimensionExhausted": {
                "memory": 1
            },
            "NodesAvailable": {
                "dc1": 1
            },
            "NodesEvaluated": 1,
            "NodesExhausted": 1,
            "NodesFiltered": 0,
            "QuotaExhausted": null,
            "ResourcesExhausted": {
                "redis": {
                    "MemoryMB": 50000
                }
            }
        }
    },
    "ID": "8057d3dd-a659-24da-e746-9dd3fffafefd",
    "JobID": "example",
    "Status": "blocked",
...

That NodesAvailable: {dc1: 1} makes me think we should also see a prometheus metric like:

nomad_nomad_blocked_evals_memory{datacenter="dc1",node_class=""} 5000

Due to the code here: https://github.com/hashicorp/nomad/blob/v1.3.2/nomad/blocked_evals_stats.go#L102-L108

Not sure where things are falling apart, so reopening this for investigation.

[github-actions]

I'm going to lock this issue because it has been closed for 120 days ⏳. This helps our maintainers find and focus on the active issues.
If you have found a problem that seems similar to this, please open a new issue and complete the issue template so we can capture all the details necessary to investigate further.