Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

metrics: classless blocked evals get metrics #13786

Merged
merged 1 commit into from
Jul 18, 2022

Conversation

shoenig
Copy link
Member

@shoenig shoenig commented Jul 15, 2022

This PR fixes a bug where blocked evaluations with no class set would
not have metrics exported at the dc:class scope.

Fixes #13759

Also drive-by fixup naming of ByNode to ByClassInDC which is more accurate 🤦

Backport to 1.3.x, 1.2.x

@shoenig
Copy link
Member Author

shoenig commented Jul 15, 2022

Spot check

➜ nomad eval status 33
ID                 = 33e76d40
Create Time        = 9s ago
Modify Time        = 9s ago
Status             = blocked
Status Description = created to place remaining allocations
Type               = service
TriggeredBy        = queued-allocs
Job ID             = example
Namespace          = default
Priority           = 50
Placement Failures = N/A - In Progress

Failed Placements
Task Group "cache" (failed to place 1 allocation):
  * Resources exhausted on 1 nodes
  * Dimension "memory" exhausted on 1 nodes
➜ napi /v1/metrics?format=prometheus | rg -v '^#' | rg blocked_eval 
nomad_nomad_blocked_evals_cpu{datacenter="dc1",host="x52",node_class=""} 0
nomad_nomad_blocked_evals_job_cpu{host="x52",job="example",namespace="default"} 0
nomad_nomad_blocked_evals_job_memory{host="x52",job="example",namespace="default"} 64000
nomad_nomad_blocked_evals_memory{datacenter="dc1",host="x52",node_class=""} 64000
nomad_nomad_blocked_evals_total_blocked{host="x52"} 1
nomad_nomad_blocked_evals_total_escaped{host="x52"} 0
nomad_nomad_blocked_evals_total_quota_limit{host="x52"} 0

@shoenig shoenig changed the title metrics: even classless blocked evals get metrics metrics: classless blocked evals get metrics Jul 15, 2022
@shoenig shoenig added backport/1.2.x backport to 1.1.x release line backport/1.3.x backport to 1.3.x release line labels Jul 15, 2022
This PR fixes a bug where blocked evaluations with no class set would
not have metrics exported at the dc:class scope.

Fixes #13759
@shoenig shoenig force-pushed the b-metrics-for-classless-blocked-evals branch from a409ac0 to 582a8a9 Compare July 15, 2022 19:13
@shoenig shoenig marked this pull request as ready for review July 15, 2022 19:27
@shoenig shoenig requested review from schmichael and lgfa29 July 15, 2022 19:27
@@ -89,6 +89,10 @@ func generateResourceStats(eval *structs.Evaluation) *BlockedResourcesStats {
for class := range allocMetrics.ClassExhausted {
classes[class] = struct{}{}
}
if len(allocMetrics.ClassExhausted) == 0 {
// some evaluations have no class
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Rude 😂

Copy link
Member

@schmichael schmichael left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks Seth!

@github-actions
Copy link

I'm going to lock this pull request because it has been closed for 120 days ⏳. This helps our maintainers find and focus on the active contributions.
If you have found a problem that seems related to this change, please open a new issue and complete the issue template so we can capture all the details necessary to investigate further.

@github-actions github-actions bot locked as resolved and limited conversation to collaborators Nov 16, 2022
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Labels
backport/1.2.x backport to 1.1.x release line backport/1.3.x backport to 1.3.x release line
Projects
None yet
Development

Successfully merging this pull request may close these issues.

When there are blocked evaluations, nomad.nomad.blocked_evals.[cpu,memory] are always 0
3 participants