Skip to content
This repository was archived by the owner on Feb 3, 2021. It is now read-only.

Calling start-shuffle-service.sh script is not necessary and can cause worker registration to fail #491

Closed
jafreck opened this issue Apr 6, 2018 · 1 comment
Labels
Milestone

Comments

@jafreck
Copy link
Member

jafreck commented Apr 6, 2018

https://spark.apache.org/docs/latest/job-scheduling.html#configuration-and-setup

"In standalone mode, simply start your workers with spark.shuffle.service.enabled set to true."

It seems the external shuffle service is built in to standalone mode. So only the spark.shuffle.service.enabled spark-defaults.conf is required. The script does not need to be run and seems to occasionally cause worker registration to fail as the workers and the external shuffle service both try to listen to port 7337.

@jafreck
Copy link
Member Author

jafreck commented Apr 9, 2018

Fixed by #492

@jafreck jafreck closed this as completed Apr 9, 2018
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Labels
Projects
None yet
Development

No branches or pull requests

1 participant