Implement spreading allocations based on a target node attribute #4527

preetapan · 2018-07-23T16:24:25Z

This PR implements support for spreading allocations across a given target node attribute (datacenter, rack etc) in the scheduler.

Spread can be configured at the task group level, or inherited from the job for all task groups. Spread can be combined with affinities, and given weights.

Builds on top of #4513 and #4512, note to reviewers - its easier if you review this after those two PRs have been merged (should happen sometime this week).

Open questions:

Allowing empty spread targets - should that infer available values for the attribute (eg, dc) at schedule time and then assume even split across all?
Interpretation of the "percent" field in each spread target, currently its normalized by dividing by the sum across all targets. This means if there is a single target with a spread percentage of 50, it really means it gets 100% of all allocations. We can make the tradeoff of requiring explicit configuration (ie enforcing that the percentages must add up to 100%) instead of doing normalization, to avoid this issue

dadgar · 2018-07-24T18:15:04Z

nomad/structs/structs.go

@@ -2185,6 +2189,13 @@ func (j *Job) Validate() error {
 		}
 	}

+	for idx, spread := range j.Spreads {


Should error if specifying it on system job?

dadgar · 2018-07-24T18:15:36Z

nomad/structs/structs.go

@@ -5384,6 +5407,82 @@ func (a *Affinity) Validate() error {
 	return mErr.ErrorOrNil()
 }

+type Spread struct {


dadgar · 2018-07-24T18:16:17Z

nomad/structs/structs.go

+	str          string
+}
+
+type SpreadTarget struct {


Don't intermix struct definitions and methods. Move SpreadTarget and its methods either below or above Spread

dadgar · 2018-07-24T18:17:25Z

nomad/structs/structs.go

+	if s.str != "" {
+		return s.str
+	}
+	s.str = fmt.Sprintf("%s %v", s.Value, s.Percent)


dadgar · 2018-07-24T18:18:41Z

nomad/structs/structs.go

+
+type SpreadTarget struct {
+	Value   string
+	Percent uint32


Validate percent is 0-100? Should we also validate that the sum of these isn't greater than 100?

dadgar · 2018-07-24T20:39:50Z

scheduler/propertyset.go

@@ -110,7 +124,7 @@ func (p *propertySet) setConstraint(constraint *structs.Constraint, taskGroup st

 // populateExisting is a helper shared when setting the constraint to populate
 // the existing values.
-func (p *propertySet) populateExisting(constraint *structs.Constraint) {
+func (p *propertySet) populateExisting(targetAttribute string) {


Why does this take the targetAttribute? Just read off of p.targetAttribute

dadgar · 2018-07-24T20:42:09Z

scheduler/spread.go

+	job               *structs.Job
+	tg                *structs.TaskGroup
+	jobSpreads        []*structs.Spread
+	tgSpreadInfo      map[string]spreadAttributeMap


Can you put some comments on these non-obvious fields and on the types

dadgar · 2018-07-24T20:43:56Z

scheduler/spread.go

+		}
+	}
+
+	// Check if there is a distinct property


Wrong comment

dadgar · 2018-07-24T20:45:54Z

scheduler/spread.go

+	combinedSpreads = append(combinedSpreads, tg.Spreads...)
+	combinedSpreads = append(combinedSpreads, iter.jobSpreads...)
+	for _, spread := range combinedSpreads {
+		sumWeight := uint32(0)


sumPercents?

dadgar · 2018-07-24T20:57:28Z

scheduler/spread.go

+			nValue, errorMsg, usedCount := pset.UsedCount(option.Node, tgName)
+			// Skip if there was errors in resolving this attribute to compute used counts
+			if errorMsg != "" {
+				continue


dadgar · 2018-07-24T22:43:48Z

scheduler/spread.go

+			desiredCount, ok := spreadDetails.desiredCounts[nValue]
+			if !ok {
+				// Warn about missing ratio
+				iter.ctx.Logger().Printf("[WARN] sched: missing desired distribution percentage for attribute value %v in spread stanza for job %v", nValue, iter.job.ID)


I wouldn't log this since it will be noisy if the user doesn't enumerate every value

Also talked about this offline, but I don't think this behavior is right. Refer to what we talked about

jippi · 2018-07-25T06:38:09Z

nomad/structs/structs.go

@@ -3336,6 +3347,10 @@ type TaskGroup struct {
 	// Affinities can be specified at the task group level to express
 	// scheduling preferences.
 	Affinities []*Affinity
+
+	// Spread can be specified at the task group level to express spreading


Will it be possible to specify on job level and have the same behavior as update{} where it cascade / inherit down from job -> spread to job -> group -> spread?

Similar question for affinity{}

@jippi yes this and affinities both work similar to the update stanza and cascade down.

dadgar · 2018-07-27T17:48:29Z

nomad/structs/structs.go

-			mErr.Errors = append(mErr.Errors, outer)
+	if j.Type == JobTypeSystem {
+		if j.Spreads != nil {
+			mErr.Errors = append(mErr.Errors, fmt.Errorf("System jobs may not have a s stanza"))


a s -> a spread

dadgar · 2018-07-27T17:49:10Z

nomad/structs/structs.go

-	Weight       int
+	// Attribute is the node attribute used as the spread criteria
+	Attribute string
+	// Weight is the relative weight of this spread, useful when there are multiple


Blank line between fields when you add comments

dadgar · 2018-07-27T17:50:20Z

nomad/structs/structs.go

+		sumPercent += target.Percent
+	}
+	if sumPercent > 100 {
+		mErr.Errors = append(mErr.Errors, errors.New("Sum of spread target percentages must not be greater than 100"))


should we make it 100%?

Sum of spread target percentages must not be greater than 100%; got %d%%

dadgar · 2018-07-27T17:50:37Z

nomad/structs/structs.go

+// for each attribute value
+type SpreadTarget struct {
+	// Value is a single attribute value, like "dc1"
+	Value string


Same comment here about spacing

dadgar · 2018-07-27T17:50:56Z

nomad/structs/structs.go

 	}
 	return mErr.ErrorOrNil()
 }

+// SpreadTarget is used to specify desired percentages


80 width lines? Why is this so truncated

dadgar · 2018-07-27T18:26:27Z

scheduler/spread.go

+					desiredCount, ok = spreadDetails.desiredCounts[ImplicitTarget]
+					if !ok {
+						// The desired count for this attribute is zero if it gets here
+						// don't boost the score


Should the score be negative here? If I have specified targets that add up to 100% and this attribute isn't there, the operator is saying it shouldn't be selected?

dadgar · 2018-07-27T18:26:53Z

scheduler/spread.go

+				}
+				if float64(usedCount) < desiredCount {
+					// Calculate the relative weight of this specific spread attribute
+					spreadWeight := float64(spreadDetails.weight) / float64(iter.sumSpreadWeights)


Add whitespace to improve readability

dadgar · 2018-07-27T18:28:17Z

scheduler/spread.go

+						continue
+					}
+				}
+				if float64(usedCount) < desiredCount {


Why is this only boosting the score? Shouldn't we add a penalty when you are > than desired count?

dadgar · 2018-07-27T21:49:50Z

scheduler/spread.go

+	// Get the nodes property value
+	nValue, ok := getProperty(option, pset.targetAttribute)
+	currentAttributeCount := uint64(0)
+	if ok {


If !ok shouldn't we return a negative score?

good catch, will fix.
i conflated !ok with the case where that specific value is not present in the combinedUse map.

dadgar · 2018-07-27T21:50:18Z

scheduler/spread.go

+	}
+	if currentAttributeCount < minCount {
+		// Small positive boost for attributes with min count
+		return evenSpreadBoost


This should be a function of the discrepancy

dadgar · 2018-07-27T21:52:14Z

scheduler/spread.go

-						// don't boost the score
-						continue
+						// so use the maximum possible penalty for this node
+						totalSpreadScore += -1.0


You are falling through to the rest of the code at 140-149 which is incorrect

dadgar · 2018-07-27T21:56:54Z

scheduler/spread.go

 	if currentAttributeCount < minCount {
-		// Small positive boost for attributes with min count
-		return evenSpreadBoost
+		// positive boost for attributes with min count


The first two cases can be simplified to if currentAttributeCount != minCount {

dadgar · 2018-07-27T22:00:51Z

scheduler/spread.go

-			return evenSpreadBoost
+			// Maximum possible boost when there is another attribute with
+			// more allocations
+			return 1.0


Why isn't this delta based from the max? You could imagine attributes with the following alloc counts:

{a1: 2, a2: 3, a3: 4}
min = 2, max = 4.

a1 should have. a score higher than a2?

dadgar

Small changes requested but LGTM

dadgar · 2018-07-30T22:58:31Z

scheduler/spread.go

@@ -171,6 +185,9 @@ func evenSpreadScoreBoost(pset *propertySet, option *structs.Node) float64 {
 	currentAttributeCount := uint64(0)
 	if ok {
 		currentAttributeCount = combinedUseMap[nValue]
+	} else {
+		// If the attribute isn't set on the node, it should get the maximum possible penalty
+		return -1.0


nValue, ok := getProperty(option, pset.targetAttribute) if !ok { return -1 } currentAttributeCount := uint64(combinedUseMap[nValue]) // Defaults to 0 anyways

dadgar · 2018-07-30T22:59:20Z

scheduler/spread.go

-	} else if currentAttributeCount > minCount {
-		// Negative boost if attribute count is greater than minimum
+	if currentAttributeCount != minCount {
+		// Boost based on delta between current and min
 		return deltaBoost
 	} else {


if currentAttribute != minCount { return deltaBoost } else if minCount == maxCount { return -1.0 } // Penalty based on delta from max value delta := int(maxCount - minCount) deltaBoost = float64(delta) / float64(minCount) return deltaBoost

dadgar · 2018-07-30T23:02:19Z

nomad/structs/structs.go

@@ -2185,6 +2189,19 @@ func (j *Job) Validate() error {
 		}
 	}

+	if j.Type == JobTypeSystem {
+		if j.Spreads != nil {
+			mErr.Errors = append(mErr.Errors, fmt.Errorf("System jobs may not have a s stanza"))


have a s stanza?

dadgar · 2018-07-30T23:03:56Z

nomad/structs/structs.go

+		sumPercent += target.Percent
+	}
+	if sumPercent > 100 {
+		mErr.Errors = append(mErr.Errors, errors.New("Sum of spread target percentages must not be greater than 100"))


Sum of spread target percentages must not be greater than 100%; got %d%%

dadgar · 2018-07-30T23:04:43Z

scheduler/generic_sched_test.go

@@ -602,6 +602,89 @@ func TestServiceSched_JobRegister_DistinctProperty_TaskGroup_Incr(t *testing.T)
 	h.AssertEvalStatus(t, structs.EvalStatusComplete)
 }

+// Test job registration with spread configured
+func TestServiceSched_Spread(t *testing.T) {


Add one for even spread?

dadgar · 2018-07-30T23:05:56Z

scheduler/generic_sched_test.go

+		&structs.Spread{
+			Attribute: "${node.datacenter}",
+			Weight:    100,
+			SpreadTarget: []*structs.SpreadTarget{


Can we maybe make this test a subtest that takes the percentages and expectations and runs through all values asserting the right outcome? [(100, 0), (90,10), (80,20), ...]

jippi · 2018-07-31T06:45:46Z

nomad/structs/structs.go

@@ -5479,7 +5479,7 @@ func (s *Spread) Validate() error {
 		sumPercent += target.Percent
 	}
 	if sumPercent > 100 {
-		mErr.Errors = append(mErr.Errors, errors.New("Sum of spread target percentages must not be greater than 100"))
+		mErr.Errors = append(mErr.Errors, errors.New(fmt.Sprintf("Sum of spread target percentages must not be greater than 100%%; got %d%%", sumPercent)))


maybe just fmt.Errorf

preetapan · 2018-07-31T16:05:10Z

scheduler/generic_sched_test.go

+	step := uint32(10)
+
+	for i := 0; i < 10; i++ {
+		name := fmt.Sprintf("%d%% in dc1", start)


this test cycles through combinations of spread across both data centers, going from (100, 0) to (0, 100) increments of 10% at a time.

…red count in each target. Added this as a new step in the stack and some unit tests

…ired counts.

instead, calculate them based on delta between current and minimum value

Also addressed other small code review comments

github-actions · 2023-02-28T02:18:09Z

I'm going to lock this pull request because it has been closed for 120 days ⏳. This helps our maintainers find and focus on the active contributions.
If you have found a problem that seems related to this change, please open a new issue and complete the issue template so we can capture all the details necessary to investigate further.

preetapan requested a review from dadgar July 23, 2018 16:24

preetapan mentioned this pull request Jul 23, 2018

spread stanza api and parsing #4528

Merged

preetapan force-pushed the f-spread-backend branch from 40fd04c to 75dab6d Compare July 24, 2018 15:54

preetapan force-pushed the f-affinities-spread branch from 8ec5642 to ed4cb93 Compare July 24, 2018 17:41

preetapan force-pushed the f-spread-backend branch from 75dab6d to 96a67ee Compare July 24, 2018 17:49

dadgar requested changes Jul 24, 2018

View reviewed changes

jippi reviewed Jul 25, 2018

View reviewed changes

dadgar requested changes Jul 27, 2018

View reviewed changes

dadgar approved these changes Jul 30, 2018

View reviewed changes

jippi reviewed Jul 31, 2018

View reviewed changes

preetapan commented Jul 31, 2018

View reviewed changes

Preetha Appan added 16 commits July 31, 2018 19:07

Structs and validation for spread

7e9553c

Validate method, and rename ratio field to percent

7814c3b

Implement spread iterator that scores according to percentage of desi…

5fa4620

…red count in each target. Added this as a new step in the stack and some unit tests

Fix warnings

f3024ff

Include spreads configured at job level when precomputing weights/des…

090e0b7

…ired counts.

validate spread from job/task group validate methods

0b1d0b0

Allow empty spread targets, and validate target percentages.

aa621a4

fix comments

dcd329e

Support implicit spread target to account for remaining desired counts

6c9a9fb

Implement support for even spread across datacenters, with unit test

3ab38b6

Remove hardcoded boosts for even spread.

9246f92

instead, calculate them based on delta between current and minimum value

fix scoring algorithm when min count == current count

9d73c2f

comment and formatting cleanup

2f9bf27

added some unit tests for -1 spread score

402934c

more cleanup

746601a

Fix scoring logic for uneven spread to incorporate current alloc count

d8b5ec2

Also addressed other small code review comments

preetapan force-pushed the f-spread-backend branch from 8c3e316 to d8b5ec2 Compare August 1, 2018 00:08

preetapan force-pushed the f-affinities-spread branch from ed4cb93 to bb26ba3 Compare August 1, 2018 00:08

fix linting error

383a4be

preetapan merged commit a38546d into f-affinities-spread Aug 1, 2018

preetapan mentioned this pull request Sep 4, 2018

Affinities and spread #4640

Merged

github-actions bot locked as resolved and limited conversation to collaborators Feb 28, 2023

Implement spreading allocations based on a target node attribute #4527

Implement spreading allocations based on a target node attribute #4527

Conversation

preetapan commented Jul 23, 2018 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

dadgar left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

github-actions bot commented Feb 28, 2023

preetapan commented Jul 23, 2018 •

edited

Loading