Skip to content

Commit

Permalink
Support pod template for Spark 3.x applications (#2141)
Browse files Browse the repository at this point in the history 
* Update API definition to support pod template

Signed-off-by: Yi Chen <[email protected]>

* Mark pod template field as schemaless

Signed-off-by: Yi Chen <[email protected]>

* Add kubebuilder marker to preserve unknown fields

Signed-off-by: Yi Chen <[email protected]>

* Add example for using pod template

Signed-off-by: Yi Chen <[email protected]>

* Support pod template

Signed-off-by: Yi Chen <[email protected]>

---------

Signed-off-by: Yi Chen <[email protected]>
  • Loading branch information
@ChenYi015
ChenYi015 authored Oct 24, 2024
1 parent fd2e125 commit d0daf2f
Show file tree
Hide file tree
Showing 17 changed files with 566 additions and 29 deletions.
2 changes: 1 addition & 1 deletion Makefile
View file
Original file line number Diff line number Diff line change
Expand Up @@ -109,7 +109,7 @@ print-%: ; @echo $*=$($*)

.PHONY: manifests
manifests: controller-gen ## Generate CustomResourceDefinition, RBAC and WebhookConfiguration manifests.
$(CONTROLLER_GEN) crd rbac:roleName=spark-operator-controller webhook paths="./..." output:crd:artifacts:config=config/crd/bases
$(CONTROLLER_GEN) crd:generateEmbeddedObjectMeta=true rbac:roleName=spark-operator-controller webhook paths="./..." output:crd:artifacts:config=config/crd/bases

.PHONY: generate
generate: controller-gen ## Generate code containing DeepCopy, DeepCopyInto, and DeepCopyObject method implementations.
Expand Down
8 changes: 8 additions & 0 deletions api/v1beta2/sparkapplication_types.go
View file
Original file line number Diff line number Diff line change
Expand Up @@ -409,6 +409,14 @@ type Dependencies struct {
// SparkPodSpec defines common things that can be customized for a Spark driver or executor pod.
// TODO: investigate if we should use v1.PodSpec and limit what can be set instead.
type SparkPodSpec struct {
// Template is a pod template that can be used to define the driver or executor pod configurations that Spark configurations do not support.
// Spark version >= 3.0.0 is required.
// Ref: https://spark.apache.org/docs/latest/running-on-kubernetes.html#pod-template.
// +optional
// +kubebuilder:validation:Schemaless
// +kubebuilder:validation:Type:=object
// +kubebuilder:pruning:PreserveUnknownFields
Template *corev1.PodTemplateSpec `json:"template,omitempty"`
// Cores maps to `spark.driver.cores` or `spark.executor.cores` for the driver and executors, respectively.
// +optional
// +kubebuilder:validation:Minimum=1
Expand Down
5 changes: 5 additions & 0 deletions api/v1beta2/zz_generated.deepcopy.go
View file

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

31 changes: 31 additions & 0 deletions charts/spark-operator-chart/crds/sparkoperator.k8s.io_scheduledsparkapplications.yaml
View file
Original file line number Diff line number Diff line change
Expand Up @@ -4755,6 +4755,13 @@ spec:
- name
type: object
type: array
template:
description: |-
Template is a pod template that can be used to define the driver or executor pod configurations that Spark configurations do not support.
Spark version >= 3.0.0 is required.
Ref: https://spark.apache.org/docs/latest/running-on-kubernetes.html#pod-template.
type: object
x-kubernetes-preserve-unknown-fields: true
terminationGracePeriodSeconds:
description: Termination grace period seconds for the pod
format: int64
Expand Down Expand Up @@ -9512,6 +9519,13 @@ spec:
- name
type: object
type: array
template:
description: |-
Template is a pod template that can be used to define the driver or executor pod configurations that Spark configurations do not support.
Spark version >= 3.0.0 is required.
Ref: https://spark.apache.org/docs/latest/running-on-kubernetes.html#pod-template.
type: object
x-kubernetes-preserve-unknown-fields: true
terminationGracePeriodSeconds:
description: Termination grace period seconds for the pod
format: int64
Expand Down Expand Up @@ -10351,6 +10365,23 @@ spec:
May contain labels and annotations that will be copied into the PVC
when creating it. No other fields are allowed and will be rejected during
validation.
properties:
annotations:
additionalProperties:
type: string
type: object
finalizers:
items:
type: string
type: array
labels:
additionalProperties:
type: string
type: object
name:
type: string
namespace:
type: string
type: object
spec:
description: |-
Expand Down
31 changes: 31 additions & 0 deletions charts/spark-operator-chart/crds/sparkoperator.k8s.io_sparkapplications.yaml
View file
Original file line number Diff line number Diff line change
Expand Up @@ -4694,6 +4694,13 @@ spec:
- name
type: object
type: array
template:
description: |-
Template is a pod template that can be used to define the driver or executor pod configurations that Spark configurations do not support.
Spark version >= 3.0.0 is required.
Ref: https://spark.apache.org/docs/latest/running-on-kubernetes.html#pod-template.
type: object
x-kubernetes-preserve-unknown-fields: true
terminationGracePeriodSeconds:
description: Termination grace period seconds for the pod
format: int64
Expand Down Expand Up @@ -9421,6 +9428,13 @@ spec:
- name
type: object
type: array
template:
description: |-
Template is a pod template that can be used to define the driver or executor pod configurations that Spark configurations do not support.
Spark version >= 3.0.0 is required.
Ref: https://spark.apache.org/docs/latest/running-on-kubernetes.html#pod-template.
type: object
x-kubernetes-preserve-unknown-fields: true
terminationGracePeriodSeconds:
description: Termination grace period seconds for the pod
format: int64
Expand Down Expand Up @@ -10257,6 +10271,23 @@ spec:
May contain labels and annotations that will be copied into the PVC
when creating it. No other fields are allowed and will be rejected during
validation.
properties:
annotations:
additionalProperties:
type: string
type: object
finalizers:
items:
type: string
type: array
labels:
additionalProperties:
type: string
type: object
name:
type: string
namespace:
type: string
type: object
spec:
description: |-
Expand Down
31 changes: 31 additions & 0 deletions config/crd/bases/sparkoperator.k8s.io_scheduledsparkapplications.yaml
View file
Original file line number Diff line number Diff line change
Expand Up @@ -4755,6 +4755,13 @@ spec:
- name
type: object
type: array
template:
description: |-
Template is a pod template that can be used to define the driver or executor pod configurations that Spark configurations do not support.
Spark version >= 3.0.0 is required.
Ref: https://spark.apache.org/docs/latest/running-on-kubernetes.html#pod-template.
type: object
x-kubernetes-preserve-unknown-fields: true
terminationGracePeriodSeconds:
description: Termination grace period seconds for the pod
format: int64
Expand Down Expand Up @@ -9512,6 +9519,13 @@ spec:
- name
type: object
type: array
template:
description: |-
Template is a pod template that can be used to define the driver or executor pod configurations that Spark configurations do not support.
Spark version >= 3.0.0 is required.
Ref: https://spark.apache.org/docs/latest/running-on-kubernetes.html#pod-template.
type: object
x-kubernetes-preserve-unknown-fields: true
terminationGracePeriodSeconds:
description: Termination grace period seconds for the pod
format: int64
Expand Down Expand Up @@ -10351,6 +10365,23 @@ spec:
May contain labels and annotations that will be copied into the PVC
when creating it. No other fields are allowed and will be rejected during
validation.
properties:
annotations:
additionalProperties:
type: string
type: object
finalizers:
items:
type: string
type: array
labels:
additionalProperties:
type: string
type: object
name:
type: string
namespace:
type: string
type: object
spec:
description: |-
Expand Down
31 changes: 31 additions & 0 deletions config/crd/bases/sparkoperator.k8s.io_sparkapplications.yaml
View file
Original file line number Diff line number Diff line change
Expand Up @@ -4694,6 +4694,13 @@ spec:
- name
type: object
type: array
template:
description: |-
Template is a pod template that can be used to define the driver or executor pod configurations that Spark configurations do not support.
Spark version >= 3.0.0 is required.
Ref: https://spark.apache.org/docs/latest/running-on-kubernetes.html#pod-template.
type: object
x-kubernetes-preserve-unknown-fields: true
terminationGracePeriodSeconds:
description: Termination grace period seconds for the pod
format: int64
Expand Down Expand Up @@ -9421,6 +9428,13 @@ spec:
- name
type: object
type: array
template:
description: |-
Template is a pod template that can be used to define the driver or executor pod configurations that Spark configurations do not support.
Spark version >= 3.0.0 is required.
Ref: https://spark.apache.org/docs/latest/running-on-kubernetes.html#pod-template.
type: object
x-kubernetes-preserve-unknown-fields: true
terminationGracePeriodSeconds:
description: Termination grace period seconds for the pod
format: int64
Expand Down Expand Up @@ -10257,6 +10271,23 @@ spec:
May contain labels and annotations that will be copied into the PVC
when creating it. No other fields are allowed and will be rejected during
validation.
properties:
annotations:
additionalProperties:
type: string
type: object
finalizers:
items:
type: string
type: array
labels:
additionalProperties:
type: string
type: object
name:
type: string
namespace:
type: string
type: object
spec:
description: |-
Expand Down
16 changes: 16 additions & 0 deletions docs/api-docs.md
View file
Original file line number Diff line number Diff line change
Expand Up @@ -2828,6 +2828,22 @@ TODO: investigate if we should use v1.PodSpec and limit what can be set instead.
<tbody>
<tr>
<td>
<code>template</code><br/>
<em>
<a href="https://kubernetes.io/docs/reference/generated/kubernetes-api/v1.24/#podtemplatespec-v1-core">
Kubernetes core/v1.PodTemplateSpec
</a>
</em>
</td>
<td>
<em>(Optional)</em>
<p>Template is a pod template that can be used to define the driver or executor pod configurations that Spark configurations do not support.
Spark version &gt;= 3.0.0 is required.
Ref: <a href="https://spark.apache.org/docs/latest/running-on-kubernetes.html#pod-template">https://spark.apache.org/docs/latest/running-on-kubernetes.html#pod-template</a>.</p>
</td>
</tr>
<tr>
<td>
<code>cores</code><br/>
<em>
int32
Expand Down
Loading

0 comments on commit d0daf2f

Please sign in to comment.