Bug: Spark shuffle service worker registration fail #492
Conversation
| @@ -96,14 +96,11 @@ def start_spark_worker(): | |||
| print("Connecting to master with '{0}'".format(" ".join(cmd))) | |||
| call(cmd) | |||
|
|
|||
| # enable the shuffle service on all slaves | |||
| start_shuffle_service() | |||
There was a problem hiding this comment.
so are you just not starting it now?
There was a problem hiding this comment.
I talked about this in #491, but apparently for standalone mode, it is not necessary to launch the shuffle service using this script. If you just set spark.shuffle.service.enabled to true, the shuffle service is started automatically.
There was a problem hiding this comment.
When you submit a spark application with that setting (and not using the start script), the logs indicate that the external shuffle service is running.
There was a problem hiding this comment.
so do you even need to keep the method below or you just keep for later usage?
There was a problem hiding this comment.
I was keeping it in case, but I don't think it is necessary at all. I'll remove it.
…github.com:jafreck/aztk into bug/spark-shuffle-service-worker-registration-fail
#491