You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Is your feature request related to a problem or challenge? Please describe what you are trying to do. #347 adds a local shuffle reader. It uses the shuffle read URL to determine if a fetch is local or not. This works well when running as standalone processes but does not work so well in a containerized environment such as k8s where multiple executors could be running on the same host but in different containers.
Describe the solution you'd like
Why don't we just look on disk to see if the shuffle files exist locally
Is your feature request related to a problem or challenge? Please describe what you are trying to do.
#347 adds a local shuffle reader. It uses the shuffle read URL to determine if a fetch is local or not. This works well when running as standalone processes but does not work so well in a containerized environment such as k8s where multiple executors could be running on the same host but in different containers.
Describe the solution you'd like
Why don't we just look on disk to see if the shuffle files exist locally
Describe alternatives you've considered
None
Additional context
See discussion at #347 (comment)
The text was updated successfully, but these errors were encountered: