Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

core: fix blocked eval math #13104

Merged
merged 4 commits into from
May 24, 2022
Merged

core: fix blocked eval math #13104

merged 4 commits into from
May 24, 2022

Conversation

shoenig
Copy link
Member

@shoenig shoenig commented May 24, 2022

This PR fixes blocked eval resource math.

8506bd6 - the 1 line bug fix, where we now subtract off the correct eval
c3c739c - added tests around eval math

Fixes #12848

Backport to 1.3.x, 1.2.x

@shoenig shoenig added backport/1.3.x backport to 1.3.x release line backport/1.2.x backport to 1.1.x release line labels May 24, 2022
@shoenig shoenig marked this pull request as ready for review May 24, 2022 14:56
@shoenig shoenig requested review from lgfa29 and jazzyfresh May 24, 2022 14:56
@shoenig shoenig added this to the 1.4.0 milestone May 24, 2022
In the original test, the eval generator would use a random value for
the job ID, resulting in an unxercised code path for duplicate blocked
evals.
Copy link
Contributor

@lgfa29 lgfa29 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for catching this and the code improvements! Hopefully it wasn't too bad to debug 😅

I pushed a commit to improve TestBlockedEvalsStats_BlockedResources. It was using random job IDs, so it never exercised this code path...

}
}

// Copy returns a deep copy of the blocked resource stats.
func (b BlockedResourcesStats) Copy() BlockedResourcesStats {
func (b *BlockedResourcesStats) Copy() *BlockedResourcesStats {
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Would it be worth adding a new nil check on these methods since they are pointer receivers now?

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I considered this but ended up leaving it out; this object should never be nil. If it is I don't think there's any reasonable behavior other than to panic (although I guess we could do so with a better error message).

@shoenig
Copy link
Member Author

shoenig commented May 24, 2022

I think I got lucky in debugging this, one of the first log lines I added was to print the before and after values of the blocked resources even though it seems like simple math. Seeing the values change in the wrong direction was a big red flag.

@shoenig shoenig merged commit b844f30 into main May 24, 2022
@shoenig shoenig deleted the b-blocked-eval-math branch May 24, 2022 21:23
@lgfa29 lgfa29 modified the milestones: 1.4.0, 1.3.2 Aug 24, 2022
@github-actions
Copy link

I'm going to lock this pull request because it has been closed for 120 days ⏳. This helps our maintainers find and focus on the active contributions.
If you have found a problem that seems related to this change, please open a new issue and complete the issue template so we can capture all the details necessary to investigate further.

@github-actions github-actions bot locked as resolved and limited conversation to collaborators Dec 22, 2022
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Labels
backport/1.2.x backport to 1.1.x release line backport/1.3.x backport to 1.3.x release line
Projects
None yet
Development

Successfully merging this pull request may close these issues.

nomad.nomad.blocked_evals.job.cpu unexpected values
2 participants