[Numpy2] Support for `numpy==2.0.0` (and pin `iris >=3.11`) #2395

valeriupredoi · 2024-04-12T15:44:27Z

Description

This PR introduces the changes we need to support Numpy 2.0 (these changes were collected from running the tests, so they may not be 100% exhaustive, but given that we have 95% test coverage, I'd say they are 95% exhaustive; we can deal with missed cases as they come along, if any; I would not use the ruff plugin test Numpy recommends to migrate, since that will introduce a lot of style changes that we may want to make when we include ruff as our linter). Most changes are in our testing suite, but there are a couple in the source code; these are backwards compatible changes, ie all tests pass when running with numpy=1.26.4

The only test that fails (both for Numpy2 and numpy legacy) is related to the pins on pandas, that need to go since we need to use the latest pandas for environment compatibility, see #2459 ; I'll try fix that issue here, since it's related to (but not a direct consequence of) Numpy2.

Closes #2460
Closes #2459

~~iris issue to monitor SciTools/iris#6043~~

-> which that is already in iris==3.11, which leads to

TODO pin iris>=3.11

-> pinned in commits dddc083 and 858ae11

The second issue closure is more of a brush it under the carpet via a pytest xfail, the original issue that kept creeping up is #2305 and the actual issue is with Pandas, and listed in pandas-dev/pandas#57002 - we can wait on its fix (and have us test quietness via the test xfail) or @schlunma can propose a more elegant solution in a separate PR?

Historical guff

OK @ESMValGroup/technical-lead-development-team we finally have a correct set of tests with the latest Numpy=2.0.0 (stable) - don't mind the ones related to pandas, I had to unpin it to gte pandas=2.2.2 built with Numpy 2.0 (otherwise the environment wouldn't solve) -> we have some work to do 👍 https://github.com/ESMValGroup/ESMValCore/actions/runs/9582595825/job/26421917358?pr=2395

~~Numpy 2.0 is impending~~: it is now fully out (see https://anaconda.org/conda-forge/numpy ); resources https://anaconda.org/conda-forge/numpy/files and https://github.com/numpy/numpy/releases - they recommend a test with the RC and list the changes roadmap in https://numpy.org/devdocs/numpy_2_0_migration_guide.html so we can test ourselves. Note: this branch should not be merged! If everything works out fine, we'll just pick up numpy 2.0 as per normal. ATTN: @ESMValGroup/technical-lead-development-team

seems the 2.0.0rc1 is a bit of a ninja - in Where is numpy=2.0.0rc1 on conda-forge? numpy/numpy#26271 it is reported that the channel is conda-forge/label/numpy_dev
moving to said conda channel results in conflicts for python-stratify, shapely, and pandas, since most prob they are pinning the upper bound of numpy to <2.0, see GA tests
a manual install is then needed; that results in mayhem - some of the fails are due to cftime, as @bouweandela points out in comments, see Numpy 2.0 incompatibility Unidata/cftime#318 (and linked PR), others are due to actual package compilation against a Numpy=1.x that barfs when it sees 2.x (those should be solved at recompile/redeploy time), and others are due to netCDF4 complaining about dtypes - that needs a closer look
[🧪][2] The new functionality is relevant and scientifically sound
[🛠][1] This pull request has a descriptive title and labels
[🛠][1] Changes are backward compatible
[🛠][1] Any changed dependencies have been added or removed correctly
[🛠][1] All checks below this pull request were successful

codecov · 2024-04-12T15:45:20Z

Codecov Report

All modified and coverable lines are covered by tests ✅

Project coverage is 94.66%. Comparing base (f74a2ad) to head (24f2549).
Report is 1 commits behind head on main.

Additional details and impacted files

@@            Coverage Diff             @@
##             main    #2395      +/-   ##
==========================================
- Coverage   94.76%   94.66%   -0.11%     
==========================================
  Files         251      251              
  Lines       14280    14285       +5     
==========================================
- Hits        13533    13523      -10     
- Misses        747      762      +15

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

valeriupredoi · 2024-04-12T16:23:21Z

UGH! This doesn't look good at all https://github.com/ESMValGroup/ESMValCore/actions/runs/8664961338/job/23762564320

bouweandela · 2024-04-15T08:37:18Z

xref Unidata/cftime#318

valeriupredoi · 2024-04-15T10:30:16Z

Spot on @bouweandela 🍻 Fortunately there is a mature PR for it Unidata/cftime#319

valeriupredoi · 2024-04-22T12:34:53Z

K then, just waiting for a new cftime to be released

valeriupredoi · 2024-05-27T13:11:08Z

still royally hosed even with rc2

valeriupredoi · 2024-11-11T15:10:14Z

@schlunma would you have a bit of time and come back to finishing the review on this, please, bud? Iris 3.11 is fully out and tested for here, and once we get this into main, then I can finish #2566 and with that we'll have support for the latest Python 3.13 too - we'll shipshape and completely up to date with our dependencies 🍺

valeriupredoi · 2024-11-11T15:11:03Z

also CC @bouweandela for a final review and merge, after Manu is happy 🍺

esmvalcore/preprocessor/_regrid_esmpy.py

pyproject.toml

environment.yml

tests/unit/preprocessor/_compare_with_refs/test_compare_with_refs.py

schlunma · 2024-11-11T15:43:43Z

esmvalcore/preprocessor/_regrid_unstructured.py

+        # explicitly casting to int32 since without it, in Numpy 2.0
+        # one gets OverflowError: Python integer 360 out of bounds for int8
+        # see notes https://numpy.org/devdocs/numpy_2_0_migration_guide.html
+        src_points_with_convex_hull[-2 * n_hull:-n_hull, 1] -= np.int32(360)
+        src_points_with_convex_hull[-n_hull:, 1] += np.int32(360)


Sorry, I still don't understand this. When I run the tests (numpy < 2), src_points_with_convex_hull is an array with dtype float64 as expected by the CMOR standards. Numpy >= 2 should NOT change that. It would be wrong if this array is of dtype int.

I just tested this locally with dummy arrays and it works with 360 and 360.0 (the latter would be "more" correct). Maybe this was just a problem of an early numpy version. Would it be able to test this again?

Co-authored-by: Manuel Schlund <[email protected]>

…efs.py Co-authored-by: Manuel Schlund <[email protected]>

schlunma

Thanks V, great that we will have support for numpy 2 soon! A couple of comments from me.

valeriupredoi · 2024-11-11T15:51:16Z

many thanks Manu! Let me try change back the syntax in esmvalcore/preprocessor/_regrid_unstructured.py

valeriupredoi · 2024-11-11T16:06:30Z

OK for some reason my two comments are not showing up on the main page - was saying that that test still fails with numpy=2.1.3 (latest), see test fail - numpy is looking at 360 as an int8 and spitting an Overflow, hence my workaround to make it int32 and avoid the Overflow - it could be an float64 like any coord point in CMOR - just anything to avoid silly numpy thinking it's int8

valeriupredoi · 2024-11-11T16:43:39Z

not just there yet Manu - numpy 2 is a bit of an 🫏 😁

valeriupredoi · 2024-11-11T16:52:52Z

green green green 🟢 like at WEC races 🏁 - cheers for fixing, Manu!

schlunma

All green now 🟢 Thanks V!!

valeriupredoi · 2024-11-11T16:55:01Z

I'll merge now, if that's OK with you - will get @bouweandela to review the Python 3.13 PR #2566

valeriupredoi added 3 commits April 12, 2024 16:38

add numpy 2 from conda-forge dev

8930af0

temp remove PyPI numpy - add numpy 2 from conda-forge dev

a3e0632

run a GA

233eb67

valeriupredoi added the testing label Apr 12, 2024

valeriupredoi requested review from bouweandela and schlunma April 12, 2024 15:44

valeriupredoi added 2 commits April 12, 2024 17:13

remove numpy from env

3687ea0

manually install numpy 2.0

bf8dff5

valeriupredoi added 5 commits April 15, 2024 12:31

correct conda forge channel

29284bc

correct conda forge channel

23429bc

do not install manually numpy, like a peasant

50910b4

revert to manual installation

4d5d1eb

comment out numpy gubbins i environment yml

9fb584a

valeriupredoi closed this Apr 19, 2024

valeriupredoi reopened this Apr 19, 2024

valeriupredoi added 2 commits May 7, 2024 14:07

Merge branch 'main' into numpy2-dev

91a87ed

Merge branch 'main' into numpy2-dev

6eb3d00

valeriupredoi mentioned this pull request May 10, 2024

Pin numpy<2.0.0 to avoid pulling 2.0.0rcX #2415

Merged

4 tasks

valeriupredoi added 3 commits May 13, 2024 13:09

Merge branch 'main' into numpy2-dev

ade96ca

Merge branch 'main' into numpy2-dev

ac75b84

deactivate direct download

d36350a

valeriupredoi changed the title ~~[DNM][Test Only] Test with numpy==2.0.0rc1~~ [DNM][Test Only] Test with numpy==2.0.0rc2 May 27, 2024

valeriupredoi added 2 commits May 27, 2024 13:51

use direct download again

5dfa45a

use direct download again

baedee8

valeriupredoi added 4 commits November 11, 2024 14:45

Merge branch 'main' into numpy2-dev

ef6c154

pin iris

dddc083

pin iris

858ae11

rerun GA tests see all is fine

e636c7b

valeriupredoi changed the title ~~[Numpy2] Support for numpy==2.0.0~~ [Numpy2] Support for numpy==2.0.0 (and pin iris >=3.11) Nov 11, 2024

valeriupredoi added 5 commits November 11, 2024 15:16

unrun GA

acf7838

change inline comment

801b7d0

change inline comment

bab038d

cleanup

d59d606

cleanup

b81db5f

schlunma reviewed Nov 11, 2024

View reviewed changes

valeriupredoi and others added 4 commits November 11, 2024 15:47

Update esmvalcore/preprocessor/_regrid_esmpy.py

3bd0468

Co-authored-by: Manuel Schlund <[email protected]>

Update pyproject.toml

19d7700

Co-authored-by: Manuel Schlund <[email protected]>

Update environment.yml

8adcd1b

Co-authored-by: Manuel Schlund <[email protected]>

Update tests/unit/preprocessor/_compare_with_refs/test_compare_with_r…

f11ad18

…efs.py Co-authored-by: Manuel Schlund <[email protected]>

schlunma reviewed Nov 11, 2024

View reviewed changes

revert to old syntax to recheck

e81c026

schlunma added 2 commits November 11, 2024 17:33

Don't use int8

c99f262

Restore nice error string when invalid season(s) are given

280e692

Cast 360 to original dtype

24f2549

schlunma approved these changes Nov 11, 2024

View reviewed changes

valeriupredoi merged commit 4b83308 into main Nov 11, 2024
6 checks passed

valeriupredoi deleted the numpy2-dev branch November 11, 2024 16:55

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[Numpy2] Support for `numpy==2.0.0` (and pin `iris >=3.11`) #2395

[Numpy2] Support for `numpy==2.0.0` (and pin `iris >=3.11`) #2395

valeriupredoi commented Apr 12, 2024 •

edited

Loading

codecov bot commented Apr 12, 2024 •

edited

Loading

valeriupredoi commented Apr 12, 2024

bouweandela commented Apr 15, 2024

valeriupredoi commented Apr 15, 2024

valeriupredoi commented Apr 22, 2024

valeriupredoi commented May 27, 2024

valeriupredoi commented Nov 11, 2024

valeriupredoi commented Nov 11, 2024

schlunma Nov 11, 2024

schlunma left a comment

valeriupredoi commented Nov 11, 2024

valeriupredoi commented Nov 11, 2024

valeriupredoi commented Nov 11, 2024

valeriupredoi commented Nov 11, 2024

schlunma left a comment

valeriupredoi commented Nov 11, 2024

[Numpy2] Support for numpy==2.0.0 (and pin iris >=3.11) #2395

[Numpy2] Support for numpy==2.0.0 (and pin iris >=3.11) #2395

Conversation

valeriupredoi commented Apr 12, 2024 • edited Loading

Description

codecov bot commented Apr 12, 2024 • edited Loading

Codecov Report

valeriupredoi commented Apr 12, 2024

bouweandela commented Apr 15, 2024

valeriupredoi commented Apr 15, 2024

valeriupredoi commented Apr 22, 2024

valeriupredoi commented May 27, 2024

valeriupredoi commented Nov 11, 2024

valeriupredoi commented Nov 11, 2024

schlunma Nov 11, 2024

Choose a reason for hiding this comment

schlunma left a comment

Choose a reason for hiding this comment

valeriupredoi commented Nov 11, 2024

valeriupredoi commented Nov 11, 2024

valeriupredoi commented Nov 11, 2024

valeriupredoi commented Nov 11, 2024

schlunma left a comment

Choose a reason for hiding this comment

valeriupredoi commented Nov 11, 2024

[Numpy2] Support for `numpy==2.0.0` (and pin `iris >=3.11`) #2395

[Numpy2] Support for `numpy==2.0.0` (and pin `iris >=3.11`) #2395

valeriupredoi commented Apr 12, 2024 •

edited

Loading

codecov bot commented Apr 12, 2024 •

edited

Loading