Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Fix: should not add emptyDir sizeLimit conf if it is nil #2305

Merged
merged 1 commit into from
Nov 4, 2024

Conversation

ChenYi015
Copy link
Contributor

@ChenYi015 ChenYi015 commented Nov 4, 2024

Purpose of this PR

Close #2199

Proposed changes:

  • When running spark-submit, the emptyDir sizeLimit conf should not be added if sizeLimit is nil.

Change Category

  • Bugfix (non-breaking change which fixes an issue)
  • Feature (non-breaking change which adds functionality)
  • Breaking change (fix or feature that could affect existing functionality)
  • Documentation update

Rationale

Checklist

  • I have conducted a self-review of my own code.
  • I have updated documentation accordingly.
  • I have added tests that prove my changes are effective or that my feature works.
  • Existing unit tests pass locally with my changes.

Additional Notes

@ChenYi015
Copy link
Contributor Author

/assign @jacobsalway @ImpSy

@jacobsalway
Copy link
Member

/lgtm

Tested locally on Kind to reproduce bug on master and confirmed the fix works

@google-oss-prow google-oss-prow bot added the lgtm label Nov 4, 2024
@ChenYi015
Copy link
Contributor Author

/approve

Copy link
Contributor

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: ChenYi015, Cian911

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

@jacobsalway
Copy link
Member

jacobsalway commented Nov 4, 2024

On a side note: I added the original change to support sizeLimit on emptyDir volumes so this looks like it's a bug introduced in the controller-runtime rewrite.

#1993

@google-oss-prow google-oss-prow bot merged commit 763682d into kubeflow:master Nov 4, 2024
11 checks passed
@ChenYi015 ChenYi015 deleted the fix/empty-dir branch November 4, 2024 11:18
@Cian911
Copy link
Contributor

Cian911 commented Nov 4, 2024

Amazing, thanks @ChenYi015 & @jacobsalway for this.

Is it possible you can do a new rc- release so I can better test this internally?

ChenYi015 added a commit to ChenYi015/spark-operator that referenced this pull request Dec 10, 2024
@ChenYi015 ChenYi015 mentioned this pull request Dec 10, 2024
google-oss-prow bot pushed a commit that referenced this pull request Dec 11, 2024
* Allow setting automountServiceAccountToken (#2298)

* Allow setting automountServiceAccountToken on workloads and serviceAccounts

Signed-off-by: Aran Shavit <[email protected]>

* update helm docs

Signed-off-by: Aran Shavit <[email protected]>

---------

Signed-off-by: Aran Shavit <[email protected]>
(cherry picked from commit 515d805)

* Fix: executor container security context does not work (#2306)

Signed-off-by: Yi Chen <[email protected]>
(cherry picked from commit 171e429)

* Fix: should not add emptyDir sizeLimit conf if it is nil (#2305)

Signed-off-by: Yi Chen <[email protected]>
(cherry picked from commit 763682d)

* Allow the Controller and Webhook Containers to run with the securityContext: readOnlyRootfilesystem: true (#2282)

* create a tmp dir for the controller to write Spark artifacts to and set the controller to readOnlyRootFilesystem

Signed-off-by: Nick Gretzon <[email protected]>

* mount a dir for the webhook container to generate its certificates in and set readOnlyRootFilesystem: true for the webhook pod

Signed-off-by: Nick Gretzon <[email protected]>

* update the securityContext in the controller deployment test

Signed-off-by: Nick Gretzon <[email protected]>

* update securityContext of the webhook container in the deployment_test

Signed-off-by: Nick Gretzon <[email protected]>

* update README

Signed-off-by: Nick Gretzon <[email protected]>

* remove -- so comments are not rendered in the README.md

Signed-off-by: Nick Gretzon <[email protected]>

* recreate README.md after removal of comments for volumes and volumeMounts

Signed-off-by: Nick Gretzon <[email protected]>

* make indentation for volumes and volumeMounts consistent with rest of values.yaml

Signed-off-by: Nick Gretzon <[email protected]>

* Revert "make indentation for volumes and volumeMounts consistent with rest of values.yaml"

This reverts commit dba97fc.

Signed-off-by: Nick Gretzon <[email protected]>

* fix indentation in webhook and controller deployment templates for volumes and volumeMounts

Signed-off-by: Nick Gretzon <[email protected]>

* Update charts/spark-operator-chart/values.yaml

Co-authored-by: Yi Chen <[email protected]>
Signed-off-by: Nicholas Gretzon <[email protected]>

* Update charts/spark-operator-chart/values.yaml

Co-authored-by: Yi Chen <[email protected]>
Signed-off-by: Nicholas Gretzon <[email protected]>

* Update charts/spark-operator-chart/values.yaml

Co-authored-by: Yi Chen <[email protected]>
Signed-off-by: Nicholas Gretzon <[email protected]>

* Update charts/spark-operator-chart/values.yaml

Co-authored-by: Yi Chen <[email protected]>
Signed-off-by: Nicholas Gretzon <[email protected]>

* Update charts/spark-operator-chart/templates/controller/deployment.yaml

Co-authored-by: Yi Chen <[email protected]>
Signed-off-by: Nicholas Gretzon <[email protected]>

* Update charts/spark-operator-chart/templates/controller/deployment.yaml

Co-authored-by: Yi Chen <[email protected]>
Signed-off-by: Nicholas Gretzon <[email protected]>

* Update charts/spark-operator-chart/templates/webhook/deployment.yaml

Co-authored-by: Yi Chen <[email protected]>
Signed-off-by: Nicholas Gretzon <[email protected]>

* Update charts/spark-operator-chart/templates/webhook/deployment.yaml

Co-authored-by: Yi Chen <[email protected]>
Signed-off-by: Nicholas Gretzon <[email protected]>

* add additional securityContext to the controller deployment_test.yaml

Signed-off-by: Nick Gretzon <[email protected]>

---------

Signed-off-by: Nick Gretzon <[email protected]>
Signed-off-by: Nicholas Gretzon <[email protected]>
Co-authored-by: Yi Chen <[email protected]>
(cherry picked from commit 72107fd)

* Fix: should not add emptyDir sizeLimit conf on executor pods if it is nil (#2316)

Signed-off-by: Cian Gallagher <[email protected]>
(cherry picked from commit 2999546)

* Bump `volcano.sh/apis` to 1.10.0 (#2320)

Signed-off-by: Jacob Salway <[email protected]>
(cherry picked from commit 22e4fb8)

* Truncate UI service name if over 63 characters (#2311)

* Truncate UI service name if over 63 characters

Signed-off-by: Jacob Salway <[email protected]>

* Also truncate ingress name

Signed-off-by: Jacob Salway <[email protected]>

---------

Signed-off-by: Jacob Salway <[email protected]>
(cherry picked from commit 43c1888)

* Bump aquasecurity/trivy-action from 0.28.0 to 0.29.0 (#2332)

Bumps [aquasecurity/trivy-action](https://github.com/aquasecurity/trivy-action) from 0.28.0 to 0.29.0.
- [Release notes](https://github.com/aquasecurity/trivy-action/releases)
- [Commits](aquasecurity/trivy-action@0.28.0...0.29.0)

---
updated-dependencies:
- dependency-name: aquasecurity/trivy-action
  dependency-type: direct:production
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <[email protected]>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
(cherry picked from commit 270b09e)

* Bump github.com/onsi/ginkgo/v2 from 2.20.2 to 2.22.0 (#2335)

Bumps [github.com/onsi/ginkgo/v2](https://github.com/onsi/ginkgo) from 2.20.2 to 2.22.0.
- [Release notes](https://github.com/onsi/ginkgo/releases)
- [Changelog](https://github.com/onsi/ginkgo/blob/master/CHANGELOG.md)
- [Commits](onsi/ginkgo@v2.20.2...v2.22.0)

---
updated-dependencies:
- dependency-name: github.com/onsi/ginkgo/v2
  dependency-type: direct:production
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <[email protected]>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
(cherry picked from commit 40423d5)

* The webhook-key-name command-line param isn't taking effect (#2344)

Signed-off-by: C. H. Afzal <[email protected]>
(cherry picked from commit a261523)

* Robustness to driver pod taking time to create (#2315)

* Retry after driver pod now found if recent submission

Signed-off-by: Thomas Newton <[email protected]>

* Add a test

Signed-off-by: Thomas Newton <[email protected]>

* Make grace period configurable

Signed-off-by: Thomas Newton <[email protected]>

* Update test

Signed-off-by: Thomas Newton <[email protected]>

* Add an extra test with the driver pod

Signed-off-by: Thomas Newton <[email protected]>

* Separate context to create and delete the driver pod

Signed-off-by: Thomas Newton <[email protected]>

* Tidy

Signed-off-by: Thomas Newton <[email protected]>

* Autoformat

Signed-off-by: Thomas Newton <[email protected]>

* Update error message

Signed-off-by: Thomas Newton <[email protected]>

* Add helm paramater

Signed-off-by: Thomas Newton <[email protected]>

* Update internal/controller/sparkapplication/controller.go

Co-authored-by: Yi Chen <[email protected]>
Signed-off-by: Thomas Newton <[email protected]>

* Newlines between helm tests

Signed-off-by: Thomas Newton <[email protected]>

---------

Signed-off-by: Thomas Newton <[email protected]>
Co-authored-by: Yi Chen <[email protected]>
(cherry picked from commit d815e78)

* Use NSS_WRAPPER_PASSWD instead of /etc/passwd as in spark-operator image entrypoint.sh (#2312)

Signed-off-by: Aakcht <[email protected]>
(cherry picked from commit 5dd91c4)

* Move sparkctl to cmd directory (#2347)

* Move spark-operator

Signed-off-by: Yi Chen <[email protected]>

* Move sparkctl to cmd directory

Signed-off-by: Yi Chen <[email protected]>

* Remove unnecessary app package/directory

Signed-off-by: Yi Chen <[email protected]>

---------

Signed-off-by: Yi Chen <[email protected]>
(cherry picked from commit 2375a30)

* Spark Operator Official Release v2.1.0

Signed-off-by: Yi Chen <[email protected]>

---------

Signed-off-by: Yi Chen <[email protected]>
Co-authored-by: Aran Shavit <[email protected]>
Co-authored-by: Nicholas Gretzon <[email protected]>
Co-authored-by: Cian (Keen) Gallagher <[email protected]>
Co-authored-by: Jacob Salway <[email protected]>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: C. H. Afzal <[email protected]>
Co-authored-by: Thomas Newton <[email protected]>
Co-authored-by: Aakcht <[email protected]>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

[BUG] Pod creation creation fails on submission with invalid resource quantities
4 participants