Skip to content

Commit

Permalink
Update to use eventual disruption
Browse files Browse the repository at this point in the history
  • Loading branch information
@engedaam
engedaam committed Aug 23, 2024
1 parent 49a3fd6 commit db86e0a
Show file tree
Hide file tree
Showing 12 changed files with 156 additions and 64 deletions.
3 changes: 3 additions & 0 deletions hack/kwok/requirements.sh
View file
Original file line number Diff line number Diff line change
Expand Up @@ -15,3 +15,6 @@ for Version in $(seq 0 1); do
yqVersion="$Version" yq eval '.spec.versions[env(yqVersion)].schema.openAPIV3Schema.properties.spec.properties.template.properties.spec.properties.requirements.items.properties.key.x-kubernetes-validations += [
{"message": "label domain \"karpenter.kwok.sh\" is restricted", "rule": "self in [\"karpenter.kwok.sh/instance-cpu\", \"karpenter.kwok.sh/instance-memory\", \"karpenter.kwok.sh/instance-family\", \"karpenter.kwok.sh/instance-size\"] || !self.find(\"^([^/]+)\").endsWith(\"karpenter.kwok.sh\")"}]' -i kwok/charts/crds/karpenter.sh_nodepools.yaml
done

# Add ExampleReason in KwoK CloudProvider
yq eval '.spec.versions[0].schema.openAPIV3Schema.properties.spec.properties.disruption.properties.budgets.items.properties.reasons.items.enum += [ "ExampleReason" ]' -i kwok/charts/crds/karpenter.sh_nodepools.yaml
9 changes: 7 additions & 2 deletions kwok/apis/crds/karpenter.kwok.sh_kwoknodeclasses.yaml
View file
Original file line number Diff line number Diff line change
Expand Up @@ -3,7 +3,7 @@ apiVersion: apiextensions.k8s.io/v1
kind: CustomResourceDefinition
metadata:
annotations:
controller-gen.kubebuilder.io/version: v0.16.1
controller-gen.kubebuilder.io/version: v0.15.0
name: kwoknodeclasses.karpenter.kwok.sh
spec:
group: karpenter.kwok.sh
Expand Down Expand Up @@ -96,7 +96,12 @@ spec:
- Unknown
type: string
type:
description: type of condition in CamelCase or in foo.example.com/CamelCase.
description: |-
type of condition in CamelCase or in foo.example.com/CamelCase.
---
Many .condition.type values are consistent across resources like Available, but because arbitrary conditions can be
useful (see .node.status.conditions), the ability to deconflict is important.
The regex it matches is (dns1123SubdomainFmt/)?(qualifiedNameFmt)
maxLength: 316
pattern: ^([a-z0-9]([-a-z0-9]*[a-z0-9])?(\.[a-z0-9]([-a-z0-9]*[a-z0-9])?)*/)?(([A-Za-z0-9][-A-Za-z0-9_.]*)?[A-Za-z0-9])$
type: string
Expand Down
20 changes: 17 additions & 3 deletions kwok/charts/crds/karpenter.sh_nodeclaims.yaml
View file
Original file line number Diff line number Diff line change
Expand Up @@ -3,7 +3,7 @@ apiVersion: apiextensions.k8s.io/v1
kind: CustomResourceDefinition
metadata:
annotations:
controller-gen.kubebuilder.io/version: v0.16.1
controller-gen.kubebuilder.io/version: v0.15.0
name: nodeclaims.karpenter.sh
spec:
group: karpenter.sh
Expand Down Expand Up @@ -262,15 +262,19 @@ spec:
description: |-
TerminationGracePeriod is the maximum duration the controller will wait before forcefully deleting the pods on a node, measured from when deletion is first initiated.
Warning: this feature takes precedence over a Pod's terminationGracePeriodSeconds value, and bypasses any blocked PDBs or the karpenter.sh/do-not-disrupt annotation.
This field is intended to be used by cluster administrators to enforce that nodes can be cycled within a given time period.
When set, drifted nodes will begin draining even if there are pods blocking eviction. Draining will respect PDBs and the do-not-disrupt annotation until the TGP is reached.
Karpenter will preemptively delete pods so their terminationGracePeriodSeconds align with the node's terminationGracePeriod.
If a pod would be terminated without being granted its full terminationGracePeriodSeconds prior to the node timeout,
that pod will be deleted at T = node timeout - pod terminationGracePeriodSeconds.
The feature can also be used to allow maximum time limits for long-running jobs which can delay node termination with preStop hooks.
If left undefined, the controller will wait indefinitely for pods to be drained.
pattern: ^([0-9]+(s|m|h))+$
Expand Down Expand Up @@ -346,7 +350,12 @@ spec:
- Unknown
type: string
type:
description: type of condition in CamelCase or in foo.example.com/CamelCase.
description: |-
type of condition in CamelCase or in foo.example.com/CamelCase.
---
Many .condition.type values are consistent across resources like Available, but because arbitrary conditions can be
useful (see .node.status.conditions), the ability to deconflict is important.
The regex it matches is (dns1123SubdomainFmt/)?(qualifiedNameFmt)
maxLength: 316
pattern: ^([a-z0-9]([-a-z0-9]*[a-z0-9])?(\.[a-z0-9]([-a-z0-9]*[a-z0-9])?)*/)?(([A-Za-z0-9][-A-Za-z0-9_.]*)?[A-Za-z0-9])$
type: string
Expand Down Expand Up @@ -789,7 +798,12 @@ spec:
- Unknown
type: string
type:
description: type of condition in CamelCase or in foo.example.com/CamelCase.
description: |-
type of condition in CamelCase or in foo.example.com/CamelCase.
---
Many .condition.type values are consistent across resources like Available, but because arbitrary conditions can be
useful (see .node.status.conditions), the ability to deconflict is important.
The regex it matches is (dns1123SubdomainFmt/)?(qualifiedNameFmt)
maxLength: 316
pattern: ^([a-z0-9]([-a-z0-9]*[a-z0-9])?(\.[a-z0-9]([-a-z0-9]*[a-z0-9])?)*/)?(([A-Za-z0-9][-A-Za-z0-9_.]*)?[A-Za-z0-9])$
type: string
Expand Down
27 changes: 22 additions & 5 deletions kwok/charts/crds/karpenter.sh_nodepools.yaml
View file
Original file line number Diff line number Diff line change
Expand Up @@ -3,7 +3,7 @@ apiVersion: apiextensions.k8s.io/v1
kind: CustomResourceDefinition
metadata:
annotations:
controller-gen.kubebuilder.io/version: v0.16.1
controller-gen.kubebuilder.io/version: v0.15.0
name: nodepools.karpenter.sh
spec:
group: karpenter.sh
Expand Down Expand Up @@ -112,13 +112,16 @@ spec:
description: |-
Reasons is a list of disruption methods that this budget applies to. If Reasons is not set, this budget applies to all methods.
Otherwise, this will apply to each reason defined.
allowed reasons are Underutilized, Empty, and Drifted.
allowed reasons are Underutilized, Empty, and Drifted and additional CloudProvider-specific reasons.
items:
description: DisruptionReason defines valid reasons for disruption budgets.
description: |-
DisruptionReason defines valid reasons for disruption budgets.
CloudProviders will need to append to the list of enums when implementing cloud provider disruption reasons
enum:
- Underutilized
- Empty
- Drifted
- ExampleReason
type: string
type: array
schedule:
Expand Down Expand Up @@ -390,15 +393,19 @@ spec:
description: |-
TerminationGracePeriod is the maximum duration the controller will wait before forcefully deleting the pods on a node, measured from when deletion is first initiated.
Warning: this feature takes precedence over a Pod's terminationGracePeriodSeconds value, and bypasses any blocked PDBs or the karpenter.sh/do-not-disrupt annotation.
This field is intended to be used by cluster administrators to enforce that nodes can be cycled within a given time period.
When set, drifted nodes will begin draining even if there are pods blocking eviction. Draining will respect PDBs and the do-not-disrupt annotation until the TGP is reached.
Karpenter will preemptively delete pods so their terminationGracePeriodSeconds align with the node's terminationGracePeriod.
If a pod would be terminated without being granted its full terminationGracePeriodSeconds prior to the node timeout,
that pod will be deleted at T = node timeout - pod terminationGracePeriodSeconds.
The feature can also be used to allow maximum time limits for long-running jobs which can delay node termination with preStop hooks.
If left undefined, the controller will wait indefinitely for pods to be drained.
pattern: ^([0-9]+(s|m|h))+$
Expand Down Expand Up @@ -470,7 +477,12 @@ spec:
- Unknown
type: string
type:
description: type of condition in CamelCase or in foo.example.com/CamelCase.
description: |-
type of condition in CamelCase or in foo.example.com/CamelCase.
---
Many .condition.type values are consistent across resources like Available, but because arbitrary conditions can be
useful (see .node.status.conditions), the ability to deconflict is important.
The regex it matches is (dns1123SubdomainFmt/)?(qualifiedNameFmt)
maxLength: 316
pattern: ^([a-z0-9]([-a-z0-9]*[a-z0-9])?(\.[a-z0-9]([-a-z0-9]*[a-z0-9])?)*/)?(([A-Za-z0-9][-A-Za-z0-9_.]*)?[A-Za-z0-9])$
type: string
Expand Down Expand Up @@ -1034,7 +1046,12 @@ spec:
- Unknown
type: string
type:
description: type of condition in CamelCase or in foo.example.com/CamelCase.
description: |-
type of condition in CamelCase or in foo.example.com/CamelCase.
---
Many .condition.type values are consistent across resources like Available, but because arbitrary conditions can be
useful (see .node.status.conditions), the ability to deconflict is important.
The regex it matches is (dns1123SubdomainFmt/)?(qualifiedNameFmt)
maxLength: 316
pattern: ^([a-z0-9]([-a-z0-9]*[a-z0-9])?(\.[a-z0-9]([-a-z0-9]*[a-z0-9])?)*/)?(([A-Za-z0-9][-A-Za-z0-9_.]*)?[A-Za-z0-9])$
type: string
Expand Down
20 changes: 17 additions & 3 deletions pkg/apis/crds/karpenter.sh_nodeclaims.yaml
View file
Original file line number Diff line number Diff line change
Expand Up @@ -3,7 +3,7 @@ apiVersion: apiextensions.k8s.io/v1
kind: CustomResourceDefinition
metadata:
annotations:
controller-gen.kubebuilder.io/version: v0.16.1
controller-gen.kubebuilder.io/version: v0.15.0
name: nodeclaims.karpenter.sh
spec:
group: karpenter.sh
Expand Down Expand Up @@ -260,15 +260,19 @@ spec:
description: |-
TerminationGracePeriod is the maximum duration the controller will wait before forcefully deleting the pods on a node, measured from when deletion is first initiated.
Warning: this feature takes precedence over a Pod's terminationGracePeriodSeconds value, and bypasses any blocked PDBs or the karpenter.sh/do-not-disrupt annotation.
This field is intended to be used by cluster administrators to enforce that nodes can be cycled within a given time period.
When set, drifted nodes will begin draining even if there are pods blocking eviction. Draining will respect PDBs and the do-not-disrupt annotation until the TGP is reached.
Karpenter will preemptively delete pods so their terminationGracePeriodSeconds align with the node's terminationGracePeriod.
If a pod would be terminated without being granted its full terminationGracePeriodSeconds prior to the node timeout,
that pod will be deleted at T = node timeout - pod terminationGracePeriodSeconds.
The feature can also be used to allow maximum time limits for long-running jobs which can delay node termination with preStop hooks.
If left undefined, the controller will wait indefinitely for pods to be drained.
pattern: ^([0-9]+(s|m|h))+$
Expand Down Expand Up @@ -344,7 +348,12 @@ spec:
- Unknown
type: string
type:
description: type of condition in CamelCase or in foo.example.com/CamelCase.
description: |-
type of condition in CamelCase or in foo.example.com/CamelCase.
---
Many .condition.type values are consistent across resources like Available, but because arbitrary conditions can be
useful (see .node.status.conditions), the ability to deconflict is important.
The regex it matches is (dns1123SubdomainFmt/)?(qualifiedNameFmt)
maxLength: 316
pattern: ^([a-z0-9]([-a-z0-9]*[a-z0-9])?(\.[a-z0-9]([-a-z0-9]*[a-z0-9])?)*/)?(([A-Za-z0-9][-A-Za-z0-9_.]*)?[A-Za-z0-9])$
type: string
Expand Down Expand Up @@ -785,7 +794,12 @@ spec:
- Unknown
type: string
type:
description: type of condition in CamelCase or in foo.example.com/CamelCase.
description: |-
type of condition in CamelCase or in foo.example.com/CamelCase.
---
Many .condition.type values are consistent across resources like Available, but because arbitrary conditions can be
useful (see .node.status.conditions), the ability to deconflict is important.
The regex it matches is (dns1123SubdomainFmt/)?(qualifiedNameFmt)
maxLength: 316
pattern: ^([a-z0-9]([-a-z0-9]*[a-z0-9])?(\.[a-z0-9]([-a-z0-9]*[a-z0-9])?)*/)?(([A-Za-z0-9][-A-Za-z0-9_.]*)?[A-Za-z0-9])$
type: string
Expand Down
26 changes: 21 additions & 5 deletions pkg/apis/crds/karpenter.sh_nodepools.yaml
View file
Original file line number Diff line number Diff line change
Expand Up @@ -3,7 +3,7 @@ apiVersion: apiextensions.k8s.io/v1
kind: CustomResourceDefinition
metadata:
annotations:
controller-gen.kubebuilder.io/version: v0.16.1
controller-gen.kubebuilder.io/version: v0.15.0
name: nodepools.karpenter.sh
spec:
group: karpenter.sh
Expand Down Expand Up @@ -112,9 +112,11 @@ spec:
description: |-
Reasons is a list of disruption methods that this budget applies to. If Reasons is not set, this budget applies to all methods.
Otherwise, this will apply to each reason defined.
allowed reasons are Underutilized, Empty, and Drifted.
allowed reasons are Underutilized, Empty, and Drifted and additional CloudProvider-specific reasons.
items:
description: DisruptionReason defines valid reasons for disruption budgets.
description: |-
DisruptionReason defines valid reasons for disruption budgets.
CloudProviders will need to append to the list of enums when implementing cloud provider disruption reasons
enum:
- Underutilized
- Empty
Expand Down Expand Up @@ -388,15 +390,19 @@ spec:
description: |-
TerminationGracePeriod is the maximum duration the controller will wait before forcefully deleting the pods on a node, measured from when deletion is first initiated.
Warning: this feature takes precedence over a Pod's terminationGracePeriodSeconds value, and bypasses any blocked PDBs or the karpenter.sh/do-not-disrupt annotation.
This field is intended to be used by cluster administrators to enforce that nodes can be cycled within a given time period.
When set, drifted nodes will begin draining even if there are pods blocking eviction. Draining will respect PDBs and the do-not-disrupt annotation until the TGP is reached.
Karpenter will preemptively delete pods so their terminationGracePeriodSeconds align with the node's terminationGracePeriod.
If a pod would be terminated without being granted its full terminationGracePeriodSeconds prior to the node timeout,
that pod will be deleted at T = node timeout - pod terminationGracePeriodSeconds.
The feature can also be used to allow maximum time limits for long-running jobs which can delay node termination with preStop hooks.
If left undefined, the controller will wait indefinitely for pods to be drained.
pattern: ^([0-9]+(s|m|h))+$
Expand Down Expand Up @@ -468,7 +474,12 @@ spec:
- Unknown
type: string
type:
description: type of condition in CamelCase or in foo.example.com/CamelCase.
description: |-
type of condition in CamelCase or in foo.example.com/CamelCase.
---
Many .condition.type values are consistent across resources like Available, but because arbitrary conditions can be
useful (see .node.status.conditions), the ability to deconflict is important.
The regex it matches is (dns1123SubdomainFmt/)?(qualifiedNameFmt)
maxLength: 316
pattern: ^([a-z0-9]([-a-z0-9]*[a-z0-9])?(\.[a-z0-9]([-a-z0-9]*[a-z0-9])?)*/)?(([A-Za-z0-9][-A-Za-z0-9_.]*)?[A-Za-z0-9])$
type: string
Expand Down Expand Up @@ -1030,7 +1041,12 @@ spec:
- Unknown
type: string
type:
description: type of condition in CamelCase or in foo.example.com/CamelCase.
description: |-
type of condition in CamelCase or in foo.example.com/CamelCase.
---
Many .condition.type values are consistent across resources like Available, but because arbitrary conditions can be
useful (see .node.status.conditions), the ability to deconflict is important.
The regex it matches is (dns1123SubdomainFmt/)?(qualifiedNameFmt)
maxLength: 316
pattern: ^([a-z0-9]([-a-z0-9]*[a-z0-9])?(\.[a-z0-9]([-a-z0-9]*[a-z0-9])?)*/)?(([A-Za-z0-9][-A-Za-z0-9_.]*)?[A-Za-z0-9])$
type: string
Expand Down
Loading

0 comments on commit db86e0a

Please sign in to comment.