Add emptyDir sizeLimit support for local dirs

[jacobsalway]

Purpose of this PR

• Support the sizeLimit field for emptyDir volume types that are prefixed with spark-local-dir-...

Change Category

Indicate the type of change by marking the applicable boxes:

• Bugfix (non-breaking change which fixes an issue)
• Feature (non-breaking change which adds functionality)
• Breaking change (fix or feature that could affect existing functionality)
• Documentation update

Rationale

• If creating an emptyDir volume without a spark-local-dir-... prefix, the webhook will patch the appropriate volume onto the pod spec, but Spark will still create an emptyDir volume anyway https://github.com/apache/spark/blob/fd3f70c74ba63a0209a4a48a387048ef40380e0f/resource-managers/kubernetes/core/src/main/scala/org/apache/spark/deploy/k8s/features/LocalDirsFeatureStep.scala#L38-L40
• However if using a spark-local-dir-..., which the operator doesn't consider in the webhook and instead supplies at submission time, the sizeLimit is not carried through to the resulting volume in K8s
• It may be desirable to set a size limit if using an emptyDir volume e.g. if on an NVME node to prevent a node disk pressure condition

Checklist

Before submitting your PR, please review the following:

• I have conducted a self-review of my own code.
• I have updated documentation accordingly.
• I have added tests that prove my changes are effective or that my feature works.
• Existing unit tests pass locally with my changes.

Additional Notes

[vara-bonthu]

@jacobsalway, thanks for the PR! It would be great if you could also update the documentation to reflect this new feature. Additionally, please address the failing pre-commit checks.

[jacobsalway]

@vara-bonthu have bumped the app version (thought it only wanted me to bump the chart version but I understand the check now) and added a sizeLimit field to the volume example on the user guide. As an operator user I usually consult the that page and expected this to work by default given that it's supported in the SparkApplication CRD, so I think the example is good enough but happy to add additional docs if you prefer

[vara-bonthu]

/lgtm

[google-oss-prow]

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: jacobsalway, vara-bonthu

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

• OWNERS [vara-bonthu]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment