[Core feature] Spark plugin should use k8s plugin default configuration

[hamersaw]

Motivation: Why do you think this is important?

There are a number of configuration values available to the k8s plugin. The Spark plugin constructs a Pods (for Spark executor and workers) that contains these same configuration values. However, we do not currently set the Spark Pod configuration based on the default k8s plugin configuration options.

Goal: What should the final outcome look like, ideally?

The Spark plugin should apply default values from the k8s plugin to it's infrastructure pods.

Describe alternatives you've considered

We could leave the current implementation as it is, but that makes fine-grained configuration of the Spark plugin difficult or impossible.

We could introduce Spark specific configuration, but that may unnecessarily bloat the configuration.

Propose: Link/Inline OR Additional context

No response

Are you sure this issue hasn't been raised already?

• Yes

Have you read the Code of Conduct?

• Yes