Verify Dask scaling, customized options, and performance on Pangeo Hub

[choldgraf]

Description

@rabernat mentioned that we need to verify some important things about Dask performance and customization - we should figure out exactly what this means and then do it.

Value / benefit

Update this once we have a better idea of what this means

Implementation details

Some information about this is here: https://pangeo.io/cloud.html#dask

Tasks to complete

No response

Updates

No response

[rabernat]

This issue is a MUST HAVE for the migration of the Pangeo GCP production cluster.

We should make sure that all of the options documented on https://pangeo.io/cloud.html#dask actually work.

[sgibson91]

I'm not sure the environment variable setting is working

But everything else seems to work ok I think. I've never used dask before so I'm not sure!

[rabernat]

I've never used dask before so I'm not sure!

Would you be interested in setting up a meeting next week where I can share screen and walk you through what our workflows look like? Dask is really a superpower. It might be good for you to get a feel for how the different pieces fit together.

[sgibson91]

Thanks @rabernat!

I got this to work in the end, I think it takes some time for things to propagate through :)

[sgibson91]

Since I didn't come across any errors when carrying out these checks by hand (once I'd been patient and allowed dask to do it's thing!), I will close this issue in favour of #771 where will try to automate similar tests across all the daskhubs we run.

[choldgraf]

@rabernat I'd appreciate the opportunity to sit in on this meeting if it happens! There's still a lot about Dask i don't understand and i never really used it in my PhD

[sgibson91]

Yes, let's arrange something! I'll open an issue in team-compass

[damianavila]

Count me in!!!