SparkSubmit Connection Extras can be overridden #36151
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
|
cc @eladkal |
pateash
force-pushed
the
pateash/spark-operator-add-yarn-queue
branch
4 times, most recently
from
8337473
to
8c8d7bf
Compare
pateash
commented
Dec 11, 2023
| @@ -204,8 +213,8 @@ def _resolve_connection(self) -> dict[str, Any]: | |||
|
|
|||
| # Determine optional yarn queue from the extra field | |||
| extra = conn.extra_dejson | |||
| conn_data["queue"] = extra.get("queue") | |||
| conn_data["deploy_mode"] = extra.get("deploy-mode") | |||
| conn_data["queue"] = self._queue if self._queue else extra.get("queue") | |||
There was a problem hiding this comment.
Override the values if passed otherwise fetch from Spark's Connection Extras
pateash
force-pushed
the
pateash/spark-operator-add-yarn-queue
branch
from
8c8d7bf
to
b65d985
Compare
pateash
changed the title
pateash
changed the title
pateash
force-pushed
the
pateash/spark-operator-add-yarn-queue
branch
2 times, most recently
from
ed36a39
to
681002a
Compare
pateash
changed the title
pateash
force-pushed
the
pateash/spark-operator-add-yarn-queue
branch
2 times, most recently
from
4108233
to
38c45e6
Compare
josh-fell
approved these changes
Dec 13, 2023
pateash
force-pushed
the
pateash/spark-operator-add-yarn-queue
branch
from
38c45e6
to
f7bf58b
Compare
phanikumv
approved these changes
Dec 13, 2023
pateash
force-pushed
the
pateash/spark-operator-add-yarn-queue
branch
from
377cf34
to
f7bf58b
Compare
This reverts commit 8c8d7bf27575df69b03747688b23a1a292206c5a.
pateash
force-pushed
the
pateash/spark-operator-add-yarn-queue
branch
from
f7bf58b
to
820b2fa
Compare
84 tasks
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
closes: #35911
Description
Currently there are some arguments which are being provided using Spark Connection,
but there is no way to override them in SparkSubmitOperator and SparkSubmitHook
eg.
--queue: option specifies the YARN queue to which the application should be submitted.
--deploy-mode: option specified deploy mode client/cluster
more - https://spark.apache.org/docs/3.2.0/running-on-yarn.html
Use case/motivation
These use-cases are particularly useful in a multi-tenant environment where different users or groups have allocated resources in specific YARN queues, or want to use different deploy mode in each spark submit job which might be different from option provided in Spark Connection Extras.