Unpin numpy in Dask environment, leave dask pinned in Dask-SQL environment

[charlesbluca]

Unblocks remaining image build failures after the merging of dask-contrib/dask-sql#1314:

• unpins numpy in Dask's GPU CI environments to accomodate UCX-Py's dependency on numpy 1.23+ (Requre NumPy 1.23+ ucx-py#1028)
• leave dask pinned in Dask-SQL's GPU CI environments since it is now pinned to match rapids-dask-dependency (Pin dask to 2024.1.1 dask-contrib/dask-sql#1301)

[jakirkham]

Yeah we needed to bump to NumPy 1.23 as part of supporting Python 3.11 ( rapidsai/build-planning#3 (comment) )

Should add recently we added an upper bound to NumPy in preparation for NumPy 2 ( rapidsai/build-planning#29 ). We plan to look into relaxing this, but need time to test (and potentially fix issues that come up). So the pin gives us breathing room

[jakirkham]

Also looking upstream it appears Dask uses NumPy 1.22 with Python 3.9 tests. Idk if that is an issue for us, so mentioning just in case

Other environments seem to use NumPy 1.23 or newer (some are unconstrained)

[charlesbluca]

Also looking upstream it appears Dask uses NumPy 1.22 with Python 3.9 tests. Idk if that is an issue for us, so mentioning just in case

Yup that's essentially what this PR is aiming to work around by manually unpinning in the Dockerfile - typically these unpins are aimed at the 3.9 (or lowest minor Python version) environment where some of the core dependencies are often pinned in ways that are incompatible with RAPIDS.

Checking locally, this is sufficient to unblock the Dask builds, but I notice that there are still some issues around the dask pinning in dask-sql's builds:

https://gpuci.gpuopenanalytics.com/job/dask/job/dask-build-environment/job/branch/job/dask-build-env-main/BUILD_NAME=dask_sql,CUDA_VER=11.8.0,LINUX_VER=ubuntu20.04,PYTHON_VER=3.10,RAPIDS_VER=24.02/1058/console

With rapids-dask-dependency and dask-sql both pinned to 2024.1.1 for now, we shouldn't need to toy around with dask's pinning at all, so can make that change in this PR so we can wrap up all the build issues at once

[jakirkham]

Thanks Charles! 🙏

Makes sense

Do we want a NumPy 2 upper bound here?

[charlesbluca]

Do we want a NumPy 2 upper bound here?

I don't think we need it to unblock, as right now just loosening the explicit pinning means we should fall back on whatever upper bound gets set by RAPIDS, though it might be worth adding pinnings to the environments here on some core packages like numpy, pandas, etc. that we can align with RAPIDS pinnings as they change to make it immediately obvious when things have stopped working (versus right now when we're only alerted when conda is unable to solve some environment).

Expect to have more time to think about this post-GTC, but can file an issue here around this work for now to follow up on in coming weeks

[jakirkham]

Ok sure that makes. Happy to go with what we have here

Let's raise an issue to follow up on RAPIDS alignment pins for numpy, pandas, etc.

[charlesbluca]

Cool, did a quick write up of my initial thoughts here #89

[jakirkham]

Looks good. Thanks Charles! 🙏

Ready to merge?

[charlesbluca]

Yup! Can follow up here once the builds are rerun and succeed, thanks for the help @jakirkham!

[jakirkham]

Thanks Charles! 🙏

[charlesbluca]

Looks like that unblocked builds - thanks! 🙏🏼