Skip to content
This repository has been archived by the owner on Jan 11, 2023. It is now read-only.

Add Rescheduler addon #1780

Merged
merged 36 commits into from
Nov 18, 2017
Merged
Show file tree
Hide file tree
Changes from 32 commits
Commits
Show all changes
36 commits
Select commit Hold shift + click to select a range
ab8d33d
merge master upstream
Oct 26, 2017
dca734d
Revert "merge master upstream"
Oct 26, 2017
bae42e6
Merge remote-tracking branch 'upstream/master'
Oct 27, 2017
8840dec
Merge remote-tracking branch 'upstream/master'
Oct 28, 2017
5cf7e98
Merge remote-tracking branch 'upstream/master'
Oct 30, 2017
94ab4c1
Merge remote-tracking branch 'upstream/master'
Nov 6, 2017
bb5b2ce
Merge branch 'master' of https://github.com/Azure/acs-engine
Nov 7, 2017
bedfe00
Merge remote-tracking branch 'upstream/master'
Nov 8, 2017
02de386
merge master upstream
Oct 26, 2017
86b004e
Revert "merge master upstream"
Oct 26, 2017
981cf24
Merge branch 'master' of https://github.com/CecileRobertMichon/acs-en…
Nov 10, 2017
989de34
Merge remote-tracking branch 'upstream/master'
Nov 15, 2017
be3c624
Merge remote-tracking branch 'upstream/master'
Nov 15, 2017
b840a41
added rescheduler addon
Nov 15, 2017
9c2a09a
fix typo
Nov 15, 2017
ae8d620
merge master upstream
Oct 26, 2017
d31849e
Revert "merge master upstream"
Oct 26, 2017
afb09d7
merge master upstream
Oct 26, 2017
5bdf726
Revert "merge master upstream"
Oct 26, 2017
5cf36c1
added rescheduler addon
Nov 15, 2017
900cd02
fix typo
Nov 15, 2017
46de126
Merge branch 'rescheduler' of https://github.com/CecileRobertMichon/a…
Nov 16, 2017
b04de96
add rescheduler spec to mastervars
Nov 16, 2017
1b1bc40
add rescheduler spec to mastervars
Nov 16, 2017
0eb1314
Merge branch 'rescheduler' of https://github.com/CecileRobertMichon/a…
Nov 16, 2017
d4df8fc
added rescheduler image to KubeConfigs
Nov 16, 2017
0a4c0f3
Merge remote-tracking branch 'upstream/master' into rescheduler
Nov 16, 2017
93e58e5
changed to using new addon pattern
Nov 16, 2017
e4ef915
parametrize limits/requests
Nov 16, 2017
c0b3f35
fix types test for isReschedulerEnabled
Nov 16, 2017
e69d7b7
fix var name in test
Nov 16, 2017
35d65f0
added rescheduler limit params
Nov 17, 2017
3beedcd
fixed pr comments
Nov 17, 2017
8823cb3
removed extra line
Nov 17, 2017
083ec7d
add rescheduler regression test + added docs + fixed dashboard name i…
Nov 17, 2017
ee7e850
typo fix
Nov 17, 2017
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
35 changes: 35 additions & 0 deletions parts/kubernetesmasteraddons-kube-rescheduler-deployment.yaml
Original file line number Diff line number Diff line change
@@ -0,0 +1,35 @@
apiVersion: extensions/v1beta1
kind: Deployment
metadata:
name: rescheduler
namespace: kube-system
labels:
k8s-app: rescheduler
kubernetes.io/cluster-service: "true"
addonmanager.kubernetes.io/mode: Reconcile
spec:
replicas: 1
selector:
matchLabels:
k8s-app: rescheduler
template:
metadata:
labels:
k8s-app: rescheduler
annotations:
scheduler.alpha.kubernetes.io/critical-pod: ''
spec:
containers:
- image: <kubernetesReschedulerSpec>
name: rescheduler
resources:
requests:
cpu: <kubernetesReschedulerCPURequests>
memory: <kubernetesReschedulerCPULimit>
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This needs to be <kubernetesReschedulerMemoryRequests>

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

oops yes thank you!

limits:
cpu: <kubernetesReschedulerCPULimit>
memory: <kubernetesReschedulerMemoryLimit>
command:
- sh
- -c
- '/rescheduler'
16 changes: 16 additions & 0 deletions parts/kubernetesmastercustomdata.yml
Original file line number Diff line number Diff line change
Expand Up @@ -140,6 +140,15 @@ write_files:
content: !!binary |
MASTER_ADDON_HEAPSTER_DEPLOYMENT_B64_GZIP_STR

{{if .OrchestratorProfile.KubernetesConfig.IsReschedulerEnabled}}
- path: /etc/kubernetes/addons/kube-rescheduler-deployment.yaml
permissions: "0644"
encoding: gzip
owner: "root"
content: !!binary |
MASTER_ADDON_RESCHEDULER_DEPLOYMENT_B64_GZIP_STR
{{end}}

- path: /etc/kubernetes/addons/azure-storage-classes.yaml
permissions: "0644"
encoding: gzip
Expand Down Expand Up @@ -284,6 +293,13 @@ write_files:
sed -i "s|<kubernetesTillerCPULimit>|{{WrapAsVariable "kubernetesTillerCPULimit"}}|g" "/etc/kubernetes/addons/kube-tiller-deployment.yaml"
sed -i "s|<kubernetesTillerMemoryLimit>|{{WrapAsVariable "kubernetesTillerMemoryLimit"}}|g" "/etc/kubernetes/addons/kube-tiller-deployment.yaml"
{{end}}
{{if .OrchestratorProfile.KubernetesConfig.IsReschedulerEnabled}}
sed -i "s|<kubernetesReschedulerSpec>|{{WrapAsVariable "kubernetesReschedulerSpec"}}|g" "/etc/kubernetes/addons/kube-rescheduler-deployment.yaml"
sed -i "s|<kubernetesReschedulerCPURequests>|{{WrapAsVariable "kubernetesReschedulerCPURequests"}}|g" "/etc/kubernetes/addons/kube-rescheduler-deployment.yaml"
sed -i "s|<kubernetesReschedulerMemoryRequests>|{{WrapAsVariable "kubernetesReschedulerMemoryRequests"}}|g" "/etc/kubernetes/addons/kube-rescheduler-deployment.yaml"
sed -i "s|<kubernetesReschedulerCPULimit>|{{WrapAsVariable "kubernetesReschedulerCPULimit"}}|g" "/etc/kubernetes/addons/kube-rescheduler-deployment.yaml"
sed -i "s|<kubernetesReschedulerMemoryLimit>|{{WrapAsVariable "kubernetesReschedulerMemoryLimit"}}|g" "/etc/kubernetes/addons/kube-rescheduler-deployment.yaml"
{{end}}
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

for clarity let's unindent this {{end}} to match the `{{if``

sed -i "s|<kubeDNSServiceIP>|{{WrapAsVariable "kubeDNSServiceIP"}}|g" "/etc/kubernetes/addons/kube-dns-deployment.yaml"

{{if .OrchestratorProfile.KubernetesConfig.EnableRbac }}
Expand Down
5 changes: 5 additions & 0 deletions parts/kubernetesmastervars.t
Original file line number Diff line number Diff line change
Expand Up @@ -25,6 +25,11 @@
"kubernetesTillerMemoryRequests": "[parameters('kubernetesTillerMemoryRequests')]",
"kubernetesTillerCPULimit": "[parameters('kubernetesTillerCPULimit')]",
"kubernetesTillerMemoryLimit": "[parameters('kubernetesTillerMemoryLimit')]",
"kubernetesReschedulerSpec": "[parameters('kubernetesReschedulerSpec')]",
"kubernetesReschedulerCPURequests": "[parameters('kubernetesReschedulerCPURequests')]",
"kubernetesReschedulerMemoryRequests": "[parameters('kubernetesReschedulerMemoryRequests')]",
"kubernetesReschedulerCPULimit": "[parameters('kubernetesReschedulerCPULimit')]",
"kubernetesReschedulerMemoryLimit": "[parameters('kubernetesReschedulerMemoryLimit')]",
"kubernetesPodInfraContainerSpec": "[parameters('kubernetesPodInfraContainerSpec')]",
"kubernetesNodeStatusUpdateFrequency": "[parameters('kubernetesNodeStatusUpdateFrequency')]",
"kubernetesCtrlMgrNodeMonitorGracePeriod": "[parameters('kubernetesCtrlMgrNodeMonitorGracePeriod')]",
Expand Down
35 changes: 35 additions & 0 deletions parts/kubernetesparams.t
Original file line number Diff line number Diff line change
Expand Up @@ -225,6 +225,41 @@
},
"type": "string"
},
"kubernetesReschedulerSpec": {
{{PopulateClassicModeDefaultValue "kubernetesReschedulerSpec"}}
"metadata": {
"description": "The container spec for rescheduler."
},
"type": "string"
},
"kubernetesReschedulerCPURequests": {
{{PopulateClassicModeDefaultValue "kubernetesReschedulerCPURequests"}}
"metadata": {
"description": "Rescheduler CPU Requests."
},
"type": "string"
},
"kubernetesReschedulerMemoryRequests": {
{{PopulateClassicModeDefaultValue "kubernetesReschedulerMemoryRequests"}}
"metadata": {
"description": "Rescheduler Memory Requests."
},
"type": "string"
},
"kubernetesReschedulerCPULimit": {
{{PopulateClassicModeDefaultValue "kubernetesReschedulerCPULimit"}}
"metadata": {
"description": "Rescheduler CPU Limit."
},
"type": "string"
},
"kubernetesReschedulerMemoryLimit": {
{{PopulateClassicModeDefaultValue "kubernetesReschedulerMemoryLimit"}}
"metadata": {
"description": "Rescheduler Memory Limit."
},
"type": "string"
},
"kubernetesPodInfraContainerSpec": {
{{PopulateClassicModeDefaultValue "kubernetesPodInfraContainerSpec"}}
"metadata": {
Expand Down
4 changes: 4 additions & 0 deletions pkg/acsengine/const.go
Original file line number Diff line number Diff line change
Expand Up @@ -85,6 +85,10 @@ const (
DefaultOrchestratorName = "k8s"
// DefaultEtcdDiskSize specifies the default size for Kubernetes master etcd disk volumes in GB
DefaultEtcdDiskSize = "128"
// DefaultReschedulerImage defines the rescheduler deployment version on Kubernetes Clusters
DefaultReschedulerImage = "rescheduler:v0.3.1"
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Is the right version to use as default?

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

let's ship w/ latest

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

0.3.1 is the latest tag here

// DefaultReschedulerAddonName is the name of the rescheduler addon deployment
DefaultReschedulerAddonName = "rescheduler"
)

const (
Expand Down
29 changes: 27 additions & 2 deletions pkg/acsengine/defaults.go
Original file line number Diff line number Diff line change
Expand Up @@ -152,7 +152,7 @@ var (
// DefaultTillerAddonsConfig is the default tiller Kubernetes addon Config
DefaultTillerAddonsConfig = api.KubernetesAddon{
Name: DefaultTillerAddonName,
Enabled: pointerToBool(true),
Enabled: pointerToBool(api.DefaultTillerAddonEnabled),
Containers: []api.KubernetesContainerSpec{
{
Name: DefaultTillerAddonName,
Expand All @@ -167,7 +167,7 @@ var (
// DefaultDashboardAddonsConfig is the default kubernetes-dashboard addon Config
DefaultDashboardAddonsConfig = api.KubernetesAddon{
Name: DefaultDashboardAddonName,
Enabled: pointerToBool(true),
Enabled: pointerToBool(api.DefaultDashboardAddonEnabled),
Containers: []api.KubernetesContainerSpec{
{
Name: DefaultDashboardAddonName,
Expand All @@ -178,6 +178,21 @@ var (
},
},
}

// DefaultReschedulerAddonsConfig is the default rescheduler Kubernetes addon Config
DefaultReschedulerAddonsConfig = api.KubernetesAddon{
Name: DefaultReschedulerAddonName,
Enabled: pointerToBool(api.DefaultReschedulerAddonEnabled),
Containers: []api.KubernetesContainerSpec{
{
Name: DefaultReschedulerAddonName,
CPURequests: "10m",
MemoryRequests: "100Mi",
CPULimits: "10m",
MemoryLimits: "100Mi",
},
},
}
)

// SetPropertiesDefaults for the container Properties, returns true if certs are generated
Expand Down Expand Up @@ -227,6 +242,7 @@ func setOrchestratorDefaults(cs *api.ContainerService) {
o.KubernetesConfig.Addons = []api.KubernetesAddon{
DefaultTillerAddonsConfig,
DefaultDashboardAddonsConfig,
DefaultReschedulerAddonsConfig,
}
} else {
// For each addon, provide default configuration if user didn't provide its own config
Expand All @@ -240,6 +256,11 @@ func setOrchestratorDefaults(cs *api.ContainerService) {
// Provide default acs-engine config for Dashboard
o.KubernetesConfig.Addons = append(o.KubernetesConfig.Addons, DefaultDashboardAddonsConfig)
}
r := getAddonsIndexByName(o.KubernetesConfig.Addons, DefaultReschedulerAddonName)
if r < 0 {
// Provide default acs-engine config for Rescheduler
o.KubernetesConfig.Addons = append(o.KubernetesConfig.Addons, DefaultReschedulerAddonsConfig)
}
}
if o.KubernetesConfig.KubernetesImageBase == "" {
o.KubernetesConfig.KubernetesImageBase = cloudSpecConfig.KubernetesSpecConfig.KubernetesImageBase
Expand Down Expand Up @@ -334,6 +355,10 @@ func setOrchestratorDefaults(cs *api.ContainerService) {
if a.OrchestratorProfile.KubernetesConfig.Addons[d].IsEnabled(api.DefaultDashboardAddonEnabled) {
a.OrchestratorProfile.KubernetesConfig.Addons[d] = assignDefaultAddonVals(a.OrchestratorProfile.KubernetesConfig.Addons[d], DefaultDashboardAddonsConfig)
}
r := getAddonsIndexByName(a.OrchestratorProfile.KubernetesConfig.Addons, DefaultReschedulerAddonName)
if a.OrchestratorProfile.KubernetesConfig.Addons[r].IsEnabled(api.DefaultReschedulerAddonEnabled) {
a.OrchestratorProfile.KubernetesConfig.Addons[r] = assignDefaultAddonVals(a.OrchestratorProfile.KubernetesConfig.Addons[r], DefaultDashboardAddonsConfig)
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The 2nd arg to assignDefaultAddonVals() here should be DefaultReschedulerAddonsConfig. (I know, copy/paste + tired eye sockets...)

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes thanks for catching that

}

if "" == a.OrchestratorProfile.KubernetesConfig.EtcdDiskSizeGB {
a.OrchestratorProfile.KubernetesConfig.EtcdDiskSizeGB = DefaultEtcdDiskSize
Expand Down
56 changes: 54 additions & 2 deletions pkg/acsengine/engine.go
Original file line number Diff line number Diff line change
Expand Up @@ -121,6 +121,7 @@ var kubernetesAddonYamls = map[string]string{
"MASTER_ADDON_KUBERNETES_DASHBOARD_DEPLOYMENT_B64_GZIP_STR": "kubernetesmasteraddons-kubernetes-dashboard-deployment.yaml",
"MASTER_ADDON_AZURE_STORAGE_CLASSES_B64_GZIP_STR": "kubernetesmasteraddons-azure-storage-classes.yaml",
"MASTER_ADDON_TILLER_DEPLOYMENT_B64_GZIP_STR": "kubernetesmasteraddons-tiller-deployment.yaml",
"MASTER_ADDON_RESCHEDULER_DEPLOYMENT_B64_GZIP_STR": "kubernetesmasteraddons-kube-rescheduler-deployment.yaml",
}

var kubernetesAddonYamls15 = map[string]string{
Expand Down Expand Up @@ -547,6 +548,7 @@ func getParameters(cs *api.ContainerService, isClassicMode bool, generatorCode s
addValue(parametersMap, "kubernetesDNSMasqSpec", cloudSpecConfig.KubernetesSpecConfig.KubernetesImageBase+KubeConfigs[k8sVersion]["dnsmasq"])
addValue(parametersMap, "kubernetesExecHealthzSpec", cloudSpecConfig.KubernetesSpecConfig.KubernetesImageBase+KubeConfigs[k8sVersion]["exechealthz"])
addValue(parametersMap, "kubernetesHeapsterSpec", cloudSpecConfig.KubernetesSpecConfig.KubernetesImageBase+KubeConfigs[k8sVersion]["heapster"])
addValue(parametersMap, "kubernetesReschedulerSpec", cloudSpecConfig.KubernetesSpecConfig.KubernetesImageBase+KubeConfigs[k8sVersion]["rescheduler"])
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

use DefaultReschedulerAddonName as map key here

tillerAddon := getAddonByName(properties.OrchestratorProfile.KubernetesConfig.Addons, DefaultTillerAddonName)
c := getAddonContainersIndexByName(tillerAddon.Containers, DefaultTillerAddonName)
if c > -1 {
Expand All @@ -570,7 +572,20 @@ func getParameters(cs *api.ContainerService, isClassicMode bool, generatorCode s
if dashboardAddon.Containers[c].Image != "" {
addValue(parametersMap, "kubernetesDashboardSpec", dashboardAddon.Containers[c].Image)
} else {
addValue(parametersMap, "kubernetesDashboardSpec", cloudSpecConfig.KubernetesSpecConfig.KubernetesImageBase+KubeConfigs[k8sVersion]["dashboard"])
addValue(parametersMap, "kubernetesDashboardSpec", cloudSpecConfig.KubernetesSpecConfig.KubernetesImageBase+KubeConfigs[k8sVersion][DefaultDashboardAddonName])
Copy link
Contributor Author

@CecileRobertMichon CecileRobertMichon Nov 16, 2017

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@jackfrancis is there a reason why this wasn't using the name var? is it okay that I changed it? (1/3)

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

👍 and thanks!

}
}
reschedulerAddon := getAddonByName(properties.OrchestratorProfile.KubernetesConfig.Addons, DefaultReschedulerAddonName)
c = getAddonContainersIndexByName(reschedulerAddon.Containers, DefaultReschedulerAddonName)
if c > -1 {
addValue(parametersMap, "kubernetesReschedulerCPURequests", reschedulerAddon.Containers[c].CPURequests)
addValue(parametersMap, "kubernetesReschedulerCPULimit", reschedulerAddon.Containers[c].CPULimits)
addValue(parametersMap, "kubernetesReschedulerMemoryRequests", reschedulerAddon.Containers[c].MemoryRequests)
addValue(parametersMap, "kubernetesReschedulerMemoryLimit", reschedulerAddon.Containers[c].MemoryLimits)
if reschedulerAddon.Containers[c].Image != "" {
addValue(parametersMap, "kubernetesReschedulerSpec", dashboardAddon.Containers[c].Image)
} else {
addValue(parametersMap, "kubernetesReschedulerSpec", cloudSpecConfig.KubernetesSpecConfig.KubernetesImageBase+KubeConfigs[k8sVersion][DefaultReschedulerAddonName])
}
}
addValue(parametersMap, "kubernetesKubeDNSSpec", cloudSpecConfig.KubernetesSpecConfig.KubernetesImageBase+KubeConfigs[k8sVersion]["dns"])
Expand Down Expand Up @@ -962,6 +977,9 @@ func (t *TemplateGenerator) getTemplateFuncMap(cs *api.ContainerService) templat
if !profile.OrchestratorProfile.KubernetesConfig.IsDashboardEnabled() {
delete(addonYamls, "MASTER_ADDON_KUBERNETES_DASHBOARD_DEPLOYMENT_B64_GZIP_STR")
}
if !profile.OrchestratorProfile.KubernetesConfig.IsReschedulerEnabled() {
delete(addonYamls, "MASTER_ADDON_RESCHEDULER_DEPLOYMENT_B64_GZIP_STR")
}
for placeholder, filename := range addonYamls {
addonTextContents := getBase64CustomScript(filename)
str = strings.Replace(str, placeholder, addonTextContents, -1)
Expand Down Expand Up @@ -1175,6 +1193,8 @@ func (t *TemplateGenerator) getTemplateFuncMap(cs *api.ContainerService) templat
tC := getAddonContainersIndexByName(tillerAddon.Containers, DefaultTillerAddonName)
dashboardAddon := getAddonByName(cs.Properties.OrchestratorProfile.KubernetesConfig.Addons, DefaultDashboardAddonName)
dC := getAddonContainersIndexByName(dashboardAddon.Containers, DefaultDashboardAddonName)
reschedulerAddon := getAddonByName(cs.Properties.OrchestratorProfile.KubernetesConfig.Addons, DefaultReschedulerAddonName)
rC := getAddonContainersIndexByName(reschedulerAddon.Containers, DefaultReschedulerAddonName)
switch attr {
case "kubernetesHyperkubeSpec":
val = cs.Properties.OrchestratorProfile.KubernetesConfig.KubernetesImageBase + KubeConfigs[k8sVersion]["hyperkube"]
Expand All @@ -1191,7 +1211,7 @@ func (t *TemplateGenerator) getTemplateFuncMap(cs *api.ContainerService) templat
val = dashboardAddon.Containers[dC].Image
}
} else {
val = cloudSpecConfig.KubernetesSpecConfig.KubernetesImageBase + KubeConfigs[k8sVersion]["dashboard"]
val = cloudSpecConfig.KubernetesSpecConfig.KubernetesImageBase + KubeConfigs[k8sVersion][DefaultDashboardAddonName]
Copy link
Contributor Author

@CecileRobertMichon CecileRobertMichon Nov 16, 2017

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@jackfrancis same here (2/3)

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

👍 and thanks ditto

}
case "kubernetesDashboardCPURequests":
if dC > -1 {
Expand Down Expand Up @@ -1255,6 +1275,38 @@ func (t *TemplateGenerator) getTemplateFuncMap(cs *api.ContainerService) templat
} else {
val = ""
}
case "kubernetesReschedulerSpec":
if rC > -1 {
if reschedulerAddon.Containers[rC].Image != "" {
val = reschedulerAddon.Containers[rC].Image
}
} else {
val = cloudSpecConfig.KubernetesSpecConfig.KubernetesImageBase + KubeConfigs[k8sVersion][DefaultReschedulerAddonName]
}
case "kubernetesReschedulerCPURequests":
if rC > -1 {
val = reschedulerAddon.Containers[rC].CPURequests
} else {
val = ""
}
case "kubernetesReschedulerMemoryRequests":
if rC > -1 {
val = reschedulerAddon.Containers[rC].MemoryRequests
} else {
val = ""
}
case "kubernetesReschedulerCPULimit":
if rC > -1 {
val = reschedulerAddon.Containers[rC].CPULimits
} else {
val = ""
}
case "kubernetesReschedulerMemoryLimit":
if rC > -1 {
val = reschedulerAddon.Containers[rC].MemoryLimits
} else {
val = ""
}
case "kubernetesKubeDNSSpec":
val = cloudSpecConfig.KubernetesSpecConfig.KubernetesImageBase + KubeConfigs[k8sVersion]["dns"]
case "kubernetesPodInfraContainerSpec":
Expand Down
Loading