Robustness to driver pod taking time to create

[Tom-Newton]

Purpose of this PR

Improve reliability.
Closes: #2302

Proposed changes:

• Add a grace period for the driver pod to be created.
• Grace period is controlled by a new config option driver-pod-creation-grace-period. Default is 10 seconds.
• Add 3 new tests around reconciling submitted status, including the new grace period functionality. This is the majority of the lines changed.
• Expose the new config option in the helm chart and add a helm test

Change Category

• Bugfix (non-breaking change which fixes an issue)
• Feature (non-breaking change which adds functionality)
• Breaking change (fix or feature that could affect existing functionality)
• Documentation update

Checklist

• I have conducted a self-review of my own code.
• I have updated documentation accordingly - I think its sufficient just to add a description on the new argument on the helm chart and regenerate the helm docs.
• I have added tests that prove my changes are effective or that my feature works.
• Existing unit tests pass locally with my changes.

Additional Notes

Some logs of a real example on our prod cluster where the spark application was saved by the grace period added in this PR.
Explore-logs-2024-11-11 12_09_13.txt

[Tom-Newton]

Sorry for the direct ping. @ChenYi015 are you the right person to review this?

[Tom-Newton]

The CI failure looks unrelated

• [FAILED] [300.030 seconds]
Example SparkApplication spark-pi-python [It] Should complete successfully
/home/runner/work/spark-operator/spark-operator/test/e2e/sparkapplication_test.go:383

  [FAILED] Unexpected error:
      <*fmt.wrapError | 0xc001b63360>: 
      client rate limiter Wait returned an error: context deadline exceeded
      {
          msg: "client rate limiter Wait returned an error: context deadline exceeded",
          err: <context.deadlineExceededError>{},
      }
  occurred
  In [It] at: /home/runner/work/spark-operator/spark-operator/test/e2e/sparkapplication_test.go:386 @ 12/04/24 12:25:47.015

I'll try rebasing on latest main

[google-oss-prow]

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: ChenYi015

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

• OWNERS [ChenYi015]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

[ChenYi015]

@Tom-Newton Thanks for improving the robustness of Spark operator.
/lgtm

[Tom-Newton]

@Tom-Newton Thanks for improving the robustness of Spark operator.

Thanks for reviewing 🙂