allow configurable range of priorities >100

[tgross]

Users who want fine-grained control over job priorities combined with a large volume of jobs being submitted at once (ex. large batch workloads) have reported that having only 100 levels of priority isn't fine-grained enough. The range from 0 to 100 is arbitrary and hard-coded as a integer literal in all the various places it's used.

We can instead turn this constant into a server configuration value with a default of 100 (the default eval/job priority can be left at 50, or we could make that a configuration value as well). Because priority 0 is the lowest-priority in Nomad, this can be done without impacting backwards compatibility. It'd just be up for operators to set sensible values when they restart servers with new values.

cc @sofixa

[alessio-perugini]

👋 @tgross If you don't mind, I'd like to take care of this. 🤓
I was thinking about some open points:

1. What would be the minimum and maximum values of JobDefaultPriority and JobMaxPriority?
   Do we want to keep the current default values as a lower bound and allow only to go upper? Or we're okay to go below our default current values? (0-100)
2. Should we also change the default value of CoreJobPriority = JobMaxPriority * 2 to avoid possible int overflow?

For the second point, I can see two options:

• CoreJobPriority = math.MaxInt
• CoreJobPriority = JobMaxPriority+1

[tgross]

What would be the minimum and maximum values of JobDefaultPriority and JobMaxPriority?

The goal here is to give cluster admins greater range, so the minimum value should be the current hard-coded value. JobDefaultPriority = 50 and JobMaxPriority = 100.

It's a good catch to note that CoreJobPriority is set to the max * 2. We need to ensure that CoreJobPriority is always greater than any user-defined value. Off the top of my head I don't see any issues with setting CoreJobPriority so high as to where it'd cause overflow. But maybe out of an abundance of caution we cap priority to math.MaxInt16 during validation without changing the field value from int (which we need to keep for backwards compatibility of serialized values). That'll give us headroom for new features if we need it, too.

So we that would give us the following:

• CoreJobPriority is hard-coded to math.MaxInt16 (32767)
• JobMaxPriority defaults to 100 and can be configured in the range 100 <= JobMaxPriority < math.MaxInt16.
• JobDefaultPriority defaults to 50 and can be configured in the range 50 <= JobDefaultPriority < JobMaxPriority

Thanks @alessio-perugini! Looking forward to your PR!

[alessio-perugini]

@tgross I've created a quick draft to understand if I'm on the right path. #16084

[tgross]

Closed by #16084

[github-actions]

I'm going to lock this issue because it has been closed for 120 days ⏳. This helps our maintainers find and focus on the active issues.
If you have found a problem that seems similar to this, please open a new issue and complete the issue template so we can capture all the details necessary to investigate further.