Updating numpy creates a different number of warnings...which fails CI

[liamhuber]

There is one "true" failure when updating numpy to 1.25 (#1140) discussed in #1143. The remaining failures are because we get more warnings than expected. I updated numpy on my local machine, but I guess the warnings are coming from some interplay between numpy and other packages because these tests all still pass fine on my machine.

1. I think counting warnings might be a crappy test idea, maybe we can do something about that?
2. Someone will need to reproduce them so we can squash them. I'll try updating all tonight and take a crack at it again tomorrow.

The warning-counting-failures for reference:

======================================================================
FAIL: test_from_hdf_readonly (generic.test_datacontainer.TestDataContainer)
Reading from HDF should restore the read-only property.
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/runner/work/pyiron_base/pyiron_base/tests/generic/test_datacontainer.py", line 448, in test_from_hdf_readonly
    self.assertEqual(len(w), 0, "from_hdf on read_only DataContainer should not call _read_only_error.")
AssertionError: 2 != 0 : from_hdf on read_only DataContainer should not call _read_only_error.

======================================================================
FAIL: test_rewrite_hdf5 (generic.test_fileHDFio.TestFileHDFio) [warning handling - log and catch job_name warning]
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/runner/work/pyiron_base/pyiron_base/tests/generic/test_fileHDFio.py", line 411, in test_rewrite_hdf5
    self.assertEqual(len(w), 1)
AssertionError: 3 != 1

======================================================================
FAIL: test_rewrite_hdf5 (generic.test_fileHDFio.TestFileHDFio) [warning handling - deprecate exclude_groups]
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/runner/work/pyiron_base/pyiron_base/tests/generic/test_fileHDFio.py", line 428, in test_rewrite_hdf5
    self.assertEqual(len(w), 1)
AssertionError: 3 != 1

======================================================================
FAIL: test_rewrite_hdf5 (generic.test_fileHDFio.TestFileHDFio) [warning handling - deprecate exclude_nodes]
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/runner/work/pyiron_base/pyiron_base/tests/generic/test_fileHDFio.py", line 438, in test_rewrite_hdf5
    self.assertEqual(len(w), 1)
AssertionError: 3 != 1

----------------------------------------------------------------------

[liamhuber]

Welp, I updated my entire conda env and am still not able to reproduce these errors.

If they still persist after we resolve #1143 I am willing to dig deeper, but let's try fixing that and recreating the dependabot PR before burning a bunch of time on this.

[niklassiemer]

We could export the conda environment to recreate the exactly same environment. Also, did you check if we warn on more things in the CI?

[liamhuber]

Also, did you check if we warn on more things in the CI?

Derp. Yes, of course.

So the warnings are not easy to correlate with the tests in the output, so I can't easily tell which ones are the extra warnings being counted in these tests. However, I find the following suspicious candidates introduced with numpy 1.25.0:

/usr/share/miniconda3/envs/my-env/lib/python3.10/site-packages/h5io/_h5io.py:351: DeprecationWarning: `product` is deprecated as of NumPy 1.25.0, and will be removed in NumPy 2.0. Please use `prod` instead.
data = str(np.array(node).tobytes().decode(decoder))

/usr/share/miniconda3/envs/my-env/lib/python3.10/site-packages/h5io/_h5io.py:338: DeprecationWarning: `product` is deprecated as of NumPy 1.25.0, and will be removed in NumPy 2.0. Please use `prod` instead.
data = cast(np.array(node)[0])

/usr/share/miniconda3/envs/my-env/lib/python3.10/site-packages/h5io/_h5io.py:353: DeprecationWarning: `product` is deprecated as of NumPy 1.25.0, and will be removed in NumPy 2.0. Please use `prod` instead.
node_unicode = str(np.array(node).tobytes().decode('utf-8'))

/usr/share/miniconda3/envs/my-env/lib/python3.10/site-packages/h5py/_hl/attrs.py:104: DeprecationWarning: `product` is deprecated as of NumPy 1.25.0, and will be removed in NumPy 2.0. Please use `prod` instead.
self.create(name, data=value)

/usr/share/miniconda3/envs/my-env/lib/python3.10/site-packages/h5py/_hl/filters.py:250: DeprecationWarning: `product` is deprecated as of NumPy 1.25.0, and will be removed in NumPy 2.0. Please use `prod` instead.
chunks = guess_chunk(shape, maxshape, dtype.itemsize)

What I find weird about this is that product is not being explicitly called in any of these lines. Indeed, in h5io/_h5io.py:351 it's really just calling a bunch of numpy array methods: data = str(np.array(node).tobytes().decode(decoder))! So why is this warning cropping up?

I thought maybe that from numpy import product was being called or something. So I dug into the source files:

• When I look at h5io/_h5io.py, I don't see any evidence of this. In fact, when I ctrl+f "product" I find absolutely nothing. I have no idea what's going on.
• In h5py/_hl/attrs.py create is explicitly calling np.product, so this makes perfect sense.
• In h5py/_hl/filters.py guess_chunk is explicitly calling np.product, so this makes perfect sense.

On both the master branch and the 3.9.0 tag for the h5py/_hl files, there is no more trace of numpy.product -- it has been replaced entirely by their own product method. I just checked and 3.9.0 is available. So I guess upgrading from 3.8.0 to that will resolve these warnings!

For h5io we are already on the most recent release 1.7.0. There have been modifications to the problematic file, but when I compare the relevant chunk of code for 1.7.0 and main, they look identical. So here I don't know how or why it's throwing the warning to start with, and I don't see any reason to think updating will help.