daskhub: provide worker resource options for 16CPU/128GB nodes on GKE/EKS #3344
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This comment was marked as resolved.
This comment was marked as resolved.
GeorgianaElena
approved these changes
Nov 1, 2023
There was a problem hiding this comment.
This looks good to me @consideRatio! Thank you!
I left a few comments about adding more notes about where certain numbers are coming from, so we can trace back in the future if need be.
Related to docs: do you believe it makes sense to add a small (rather generic) note about these new instance-dependent dask worker options in https://docs.2i2c.org/about/distributions/research/#a-brief-overview-of-scalable-computing-hubs?
consideRatio
force-pushed
the
pr/dask-options
branch
from
a74ab02 to
b0f1283
Compare
|
Deployed to leap's staging for now, with a 👍 from @jbusecke who actively makes use of dask-gateway atm I'll go for a merge. |
consideRatio
force-pushed
the
pr/dask-options
branch
2 times, most recently
from
8eda3d7 to
42de740
Compare
consideRatio
force-pushed
the
pr/dask-options
branch
from
42de740 to
8ac8085
Compare
|
Rebased - I'll go for a merge at this point! |
|
🎉🎉🎉🎉 Monitor the deployment of the hubs here 👉 https://github.com/2i2c-org/infrastructure/actions/runs/6934459391 |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
All GKE/EKS clusters with daskhubs have
n2-highmem-16/r5.4xlargenodes setup for them (#2687). This makes us able to easily standardize pre-defined options to choose from for all clusters with regards to how large CPU/memory requests should be made for each pod.Review notes
daskhub's new default
The daskhub chart comes with these options by default now, adjusting to use
n2-highmem-16orr5.4xlargebased on the k8s distribution that is detected automatically.With this, jupyter-meets-the-earth no longer specifies its own custom logic.
basehub-cluster-infoConfigMap addedIt is what makes us have the
BASEHUB_K8S_DISTenvironment variable available for the hub and dask-gateway pod, where the dask-gateway pod relies on it in this PR and the hub pod got it mounted as its likely useful.The ConfigMap gets labels aligning with modern seemingly now stable helm chart best practices.
Test with
pange/base-notebook:latestat https://dask-staging.2i2c.cloudUX for users
@jbusecke I've deployed this to leap's staging hub for now, if you have feedback its most welcome!