2048 add support for numpy 2 #2050

G-D-Petrov · 2024-12-06T15:44:25Z

Reference Issues/PRs

#2048

What does this implement or fix?

There quite a few distinct changes as most of the work was in the CI and testing.
I will try to separate the different changes here:

Numpy 2 Support:
- support for new c++ interface
- small change for how == and is are evaluated
- added some more checks during normalization
- changes to tests to support new Numpy conventions (all of the changes to the pytests)
Pybind11 2.13 Support (Needed for numpy2):
- removed python 3.6 and added 3.12 (relevant changes here, here
  , and here
- Added new version of pybind11 and made changes to support the new interface:
  - changed PythonUtils to add more info about the Python executable/libs neded for the newer version of pybind11
  - Changes to support new pybind11 interfaces (relevant changes here, here, here, here, here, here, here and here
Changes for testing: (also added tickets for these):

I've also tested in against the full suite of tests incl the persistent ones (run here)
N.B.: We need to continue using the submodule on pybind11 instead of getting it from vcpkg, because the latest version of pybind11 are not available on its vcpkg distribution.
Specifically, the versions that introduce the Numpy 2 Support.

Any other comments?

Checklist

Checklist for code changes...

Have you updated the relevant docstrings, documentation and copyright notice?
Is this contribution tested against all ArcticDB's features?
Do all exceptions introduced raise appropriate error messages?
Are API changes highlighted in the PR description?
Is the PR labelled as enhancement or bug so it appears in autogenerated release notes?

.github/workflows/build.yml

cpp/CMake/PythonUtils.cmake

cpp/arcticdb/pipeline/frame_utils.hpp

python/arcticdb/util/_versions.py

python/arcticdb/version_store/_normalization.py

python/arcticdb/version_store/_store.py

python/tests/util/storage_test.py

alexowens90 · 2024-12-20T17:10:16Z

build_tooling/requirements-compatibility-py311.txt

@@ -0,0 +1,2 @@
+# Makes sure we are able to use Numpy 1
+numpy<2


So what matrix of Python (minor) and Pandas/numpy (major) versions are now covered by the CI?

The following is the matrix on Linux based on the latest run in this branch:

Python Version NumPy Version Pandas Version

Python 3.7 1.21.6 1.3.5

Python 3.8 1.24.4 2.0.3

Python 3.8 (Compat) 1.24.4 1.5.3

Python 3.9 2.0.2 2.2.3

Python 3.10 2.2.1 2.2.3

Python 3.11 2.2.1 2.2.3

Python 3.11 (Compat) 1.26.4 2.2.3

Python 3.12 2.2.1 2.2.3

There is no Numpy 2 for python 3.8 and below.
Python 3.6 backwards/forwards compat can still be tested when the persistent tests are run.

OK cool. Would be nice to have all 4 of numpy 1.x/2.x and pandas 1.x/2.x tested on all Python versions where that makes sense, but that way combinatorial explosions lie, so this seems like a good compromise

alexowens90 · 2024-12-20T17:17:12Z

cpp/arcticdb/entity/native_tensor.hpp

@@ -119,14 +119,16 @@ struct NativeTensor {
    [[nodiscard]] auto expanded_dim() const { return expanded_dim_; }
    template<typename T>
    const T *ptr_cast(size_t pos) const {
+        util::check(ptr != nullptr, "Unexpected null ptr in NativeTensor");


Why is this no longer needed?

It has been added. It is a bit of a sanity check because if there is a incompatibility of the numpy version used at runtime vs compile time, the whole numpy object is misread and this pointer is null

willdealtry · 2024-12-30T13:16:45Z

Are you sure that the encoding_version error has been properly understood? The values for the encoding_version are actually 0 (the default) and 1, even though we refer to it as version 1 and 2. It looks like it's complaining it has been given an invalid encoding version

G-D-Petrov · 2024-12-30T13:43:20Z

Are you sure that the encoding_version error has been properly understood? The values for the encoding_version are actually 0 (the default) and 1, even though we refer to it as version 1 and 2. It looks like it's complaining it has been given an invalid encoding version

@willdealtry yes, the older versions are performing this check that is checking specifically for version 0 aka 1

-Upgrade pybind in conda to 2.11.1 -Move pybind11 from submodule to vcpkg -Reinstate pybind11 git submodule -Bypass GIL check in some places Bypass nan increase ref and decreas ref as well Remove more py::none{} call Skip GIL check in C++ test Add support for numpy 2 Support numpy on conda Remove Werror for testing Fix deprecation warnings Test fix for new pybind cmake checks Try to disable FindPythonInterp and FindPythonLIbs Try to set the python vars manually remove policy Fix nan related tests Fix builds test with other Utils test try to set extension manually add endif change Test turning off the pybind11 override Tests without the PythonUtils module Try with latest pybind11 Test PYBIND11_NOPYTHON Test with PYBIND11_FINDPYTHON idk Try to set vars for both PYTHON and Python prefix Try setting both Python and PYTHON try new pythonutils test like this Add support for both Numpy 1 and 2 Remove 3.6 and fix some tests Totally remove 3.6 test with a new columns condition Test new action steps fix tests Test with a python pin for conda tests Test with 3.11 Test with newer changes Test without calling PythonUtils twice Test without PYBIND11_NO_ASSERT_GIL_HELD_INCREF_DECREF definition Add compact tests for Numpy 1 and test newer python versions test with latest cibuildwheel version test with latest cibuildwheel Test with newer cibuildwheel test with http endpoint Don't build for python 13 Disable compat tests for python 3.12 and 13 as we haven't published them yet Fix integration tests fix flaky test

… Pandas1 support and update build workflow

…Numpy2 + Pandas1 support and update build workflow" This reverts commit 494f983.

G-D-Petrov force-pushed the 2048-add-support-for-numpy-2 branch 3 times, most recently from 504a15f to 20d68e9 Compare December 16, 2024 13:51

G-D-Petrov marked this pull request as ready for review December 17, 2024 13:06

G-D-Petrov requested review from alexowens90, willdealtry and poodlewars as code owners December 17, 2024 13:06

G-D-Petrov force-pushed the 2048-add-support-for-numpy-2 branch from 8324df5 to 112ef00 Compare December 17, 2024 14:18