Implementation and test updates to support NumPy 1.25 #9011

apmasell · 2023-06-12T20:53:05Z

NumPy 1.25 has two major changes:

allow mixed signed comparisons ufuncs (the qQ->? and Qq->? implementations for >, <, >=, <=, ==, !=); this mode was not supported by Numba and an appropriate implementation is added to match the C implementation in NumPy.
the implementation of sin and cos are changed in a way that produces slightly different numeric results. This either lowers the comparison precision where these functions are used in tests or replaces them with another function (mostly exp) to avoid the differences.

NumPy 1.25 has two major changes: - allow mixed signed comparisons ufuncs (the `qQ->?` and `Qq->?` implementations for `>`, `<`, `>=`, `<=`, `==`, `!=`); this mode was not supported by Numba and an appropriate implementation is added to match the C implementation in NumPy. - the implementation of `sin` and `cos` are changed in a way that produces slightly different numeric results. This either lowers the comparison precision where these functions are used in tests or replaces them with another function (mostly `exp`) to avoid the differences.

numba/cpython/numbers.py

stuartarchibald · 2023-06-13T11:10:16Z

Thanks for the patch. RE sin/cos changes, I think this patch numpy/numpy#23925 reverts the implementation that was causing Numba's tests to fail, assuming this is backported to the NumPy 1.25 release branch, Numba will not need to make any related changes.

Changes the implementations to use `select` rather than a branching path for better optimization potential.

The corresponding change was removed from NumPy 1.25 so any numeric changes for sin/cos are also being reverted.

As titled.

gmarkall · 2023-06-27T11:07:14Z

buildscripts/incremental/setup_conda_environment.sh

@@ -30,7 +30,7 @@ conda list
 # Use conda-forge for NumPy 1.24 - at the time of writing it is not available
 # on the defaults channel.

-if [ "${NUMPY}" == "1.24" ]; then
+if [ "${NUMPY}" == "1.24" -o "${NUMPY}" == "1.25" ]; then


I think conda-forge is only needed for 1.25 these days (1.24 is in the default channels).

This raises a bit of a concern because we've been using conda-forge packages until now for 1.24. Should we switch this off for 1.25 once Anaconda packages are ready?

gmarkall · 2023-06-27T11:07:23Z

buildscripts/incremental/setup_conda_environment.sh

@@ -30,7 +30,7 @@ conda list
 # Use conda-forge for NumPy 1.24 - at the time of writing it is not available


Suggested change

# Use conda-forge for NumPy 1.24 - at the time of writing it is not available

# Use conda-forge for NumPy 1.25 - at the time of writing it is not available

gmarkall · 2023-06-27T11:07:37Z

buildscripts/incremental/setup_conda_environment.cmd

@@ -16,7 +16,7 @@ set PIP_INSTALL=pip install -q

 @rem Use conda-forge for NumPy 1.24 - at the time of writing it is not available


Suggested change

@rem Use conda-forge for NumPy 1.24 - at the time of writing it is not available

@rem Use conda-forge for NumPy 1.25 - at the time of writing it is not available

gmarkall · 2023-06-27T11:08:12Z

buildscripts/incremental/setup_conda_environment.cmd

@@ -16,7 +16,7 @@ set PIP_INSTALL=pip install -q

 @rem Use conda-forge for NumPy 1.24 - at the time of writing it is not available
 @rem on the defaults channel.
-if %NUMPY%==1.24 (set NUMPY_CHANNEL_PKG="conda-forge::numpy") else (set NUMPY_CHANNEL_PKG="numpy")
+if %NUMPY% GEQ 1.24 (set NUMPY_CHANNEL_PKG="conda-forge::numpy") else (set NUMPY_CHANNEL_PKG="numpy")


I think this should be OK with NP 1.24 in default channels now:

Suggested change

if %NUMPY% GEQ 1.24 (set NUMPY_CHANNEL_PKG="conda-forge::numpy") else (set NUMPY_CHANNEL_PKG="numpy")

if %NUMPY%==1.25 (set NUMPY_CHANNEL_PKG="conda-forge::numpy") else (set NUMPY_CHANNEL_PKG="numpy")

numba/cpython/numbers.py

gmarkall · 2023-06-27T11:10:35Z

numba/cpython/numbers.py

@@ -361,6 +361,38 @@ def int_ne_impl(context, builder, sig, args):
    return impl_ret_untracked(context, builder, sig.return_type, res)


+def int_su_cmp(op):


Perhaps:

Suggested change

def int_su_cmp(op):

def int_signed_unsigned_cmp(op):

would make it a bit easier to guess what the function does from the name? (I had to read the description in the implementation to figure out that s = signed and u = unsigned)

No problem. I had it this way to match the other comparison functions.

gmarkall · 2023-06-27T11:10:52Z

numba/cpython/numbers.py

+    return impl
+
+
+def int_us_cmp(op):


Similar to above,

Suggested change

def int_us_cmp(op):

def int_unsigned_signed_cmp(op):

gmarkall

Things mostly look good from a quick skim of the diff - there are some comments on it.

The main issue I've come across is that building locally with:

python setup.py develop

results in:

$ python runtests.py -m
RuntimeError: The new DType API is currently in an exploratory phase and should NOT be used for production code.  Expect modifications and crashes!  To experiment with the new API you must set `NUMPY_EXPERIMENTAL_DTYPE_API=1` as an environment variable.
Traceback (most recent call last):
  File "/home/gmarkall/numbadev/numba/numba/testing/__init__.py", line 29, in load_testsuite
    suite.addTests(loader.loadTestsFromName(f))
  File "/home/gmarkall/mambaforge/envs/numbanp125/lib/python3.10/unittest/loader.py", line 154, in loadTestsFromName
    module = __import__(module_name)
  File "/home/gmarkall/numbadev/numba/numba/tests/npyufunc/test_gufunc.py", line 5, in <module>
    import numpy.core.umath_tests as ut
  File "/home/gmarkall/mambaforge/envs/numbanp125/lib/python3.10/site-packages/numpy/core/umath_tests.py", line 13, in <module>
    from ._umath_tests import *
RuntimeError: cannot load _umath_tests module.

I think requiring the user to set an environment variable pertaining to NumPy to run the testsuite is a problem - can / should we skip the relevant tests if numpy.core.umath_tests fails to import?

As titled.

Switch `numpy.core.umath_tests.matrix_multiply`, which is private API, with `numpy.matmul`, which is public.

gmarkall · 2023-06-28T08:57:45Z

numba/__init__.py

@@ -142,7 +142,7 @@ def test(argv, **kwds):
    """.split() + types.__all__ + errors.__all__


-_min_llvmlite_version = (0, 41, 0)
+_min_llvmlite_version = (0, 40, 0)


Was this accidentally added?

seberg · 2023-06-28T09:11:35Z

RuntimeError: cannot load _umath_tests module.

Odd, conftest.py should set that environment variable so that you don't run into that (although, I will admit that may not be a great solution). Ping me if it remains/seems like a problem, then I will look into it.

stuartarchibald · 2023-07-11T16:32:47Z

Fixed and passing on the build farm.

Any chance you could please be more specific, which buildfarm DAG and which build ID? Thanks.

Prior to merge, this needs to pass on the buildfarm DAG numba-np-next (to test 1.25) and also on the numba_yaml DAG (to test this doesn't break anything existing). Public Azure and gpuCI will also need to pass against HEAD of this branch.

As titled.

numba/cuda/ufuncs.py

stuartarchibald · 2023-07-12T13:05:58Z

numba/tests/test_parfors.py

+        if numpy_version < (1, 25):
+            def test_impl(A):
+                min_val = np.amin(A)
+                return A - min_val
+        else:
+            def test_impl(A):
+                min_val = np.min(A)
+                return A - min_val


What is the purpose of this change? Is it an artifact of fixing the test prior to discovering that np.amin and np.min do not alias in 1.25? If so it can probably be reverted?

This needs reverting. It's from 1a1c7c1 and before new overloads were added

np.min is not available in NumPy 1.22

I think min is just an alias of amin in NumPy 1.22.x:

>>> import numpy as np >>> np.__version__ '1.22.3' >>> np.min <function amin at 0xsomeaddress>

I'm also of the view that this change needs reverting, amin is the function being tested originally and should continue to be tested invariant of how NumPy have implemented it.

As requested in review.

gmarkall · 2023-07-26T10:57:59Z

numba/tests/test_array_methods.py

@@ -30,6 +30,9 @@ def np_around_unary(val):
 def np_round_array(arr, decimals, out):
    np.round(arr, decimals, out)

+def np_round__array(arr, decimals, out):


What is the double-underscore variant of this function and test for? It looks identical to the version above - am I missing something?

There's round and round_ in NumPy, so, I'm trying to continue the pattern.

The function calls np.round though, rather than np.round_.

gmarkall · 2023-07-27T06:55:41Z

gpuci run tests

gmarkall · 2023-07-27T06:58:24Z

gpuci run tests

gmarkall · 2023-07-27T07:01:59Z

gpuci run tests

NumPy 1.25 packages are not provided for Python < 3.10 on the default conda channel, so we need to use a newer version.

gmarkall · 2023-07-27T07:14:47Z

gpuci run tests

There isn't a suitable RAPIDS image with 3.10 that keeps the versions of all other components the same, so instead we'll keep the 3.8 image and create a 3.10 environment in it.

gmarkall · 2023-07-27T07:19:44Z

gpuci run tests

gmarkall · 2023-07-27T07:41:15Z

@apmasell I hope you don't mind - I fixed the conflicts with the gpuCI setup and also fixed an issue with the config related to the Python version in conjunction with NumPy, so that this could be tested on gpuCI (rather than asking you to make changes and waiting).

As titled.

apmasell · 2023-07-27T11:16:43Z

No problem with the changes for gpuCI.

NumPy 1.25 support (PR #9011) continued

apmasell added the 2 - In Progress label Jun 12, 2023

apmasell mentioned this pull request Jun 12, 2023

NumPy 1.25 support #8698

Closed

sklam reviewed Jun 12, 2023

View reviewed changes

numba/cpython/numbers.py Outdated Show resolved Hide resolved

Improve implementation mixed sign comparison ufuncs

7db7e99

Changes the implementations to use `select` rather than a branching path for better optimization potential.

apmasell mentioned this pull request Jun 13, 2023

CUDA: Add comparison ufunc support #9007

Merged

apmasell added 2 commits June 20, 2023 15:24

Revert sin/cos changes for NumPy 1.25

b11bafb

The corresponding change was removed from NumPy 1.25 so any numeric changes for sin/cos are also being reverted.

Add NumPy 1.25 test environment

ac919ca

As titled.

apmasell marked this pull request as ready for review June 21, 2023 11:29

apmasell requested a review from stuartarchibald as a code owner June 21, 2023 11:29

apmasell added 3 - Ready for Review and removed 2 - In Progress labels Jun 21, 2023

esc added this to the Numba 0.58 RC milestone Jun 27, 2023

gmarkall reviewed Jun 27, 2023

View reviewed changes

esc added the highpriority label Jun 27, 2023

gmarkall reviewed Jun 27, 2023

View reviewed changes

numba/cpython/numbers.py Outdated Show resolved Hide resolved

gmarkall reviewed Jun 27, 2023

View reviewed changes

gmarkall requested changes Jun 27, 2023

View reviewed changes

gmarkall added 4 - Waiting on author Waiting for author to respond to review and removed 3 - Ready for Review labels Jun 27, 2023

apmasell added 3 commits June 27, 2023 16:15

Use conda-forge package only for NumPy 1.25

d7162f5

As titled.

Rename mixed signedness code and reference original

e338e22

As titled.

Remove dependency on umath_tests

5e3ff5e

Switch `numpy.core.umath_tests.matrix_multiply`, which is private API, with `numpy.matmul`, which is public.

gmarkall reviewed Jun 28, 2023

View reviewed changes

stuartarchibald added the 4 - Waiting on author Waiting for author to respond to review label Jul 11, 2023

apmasell added 3 commits July 12, 2023 08:10

Add additional tests for amin and amax

f4cfb3f

As titled.

Merge remote-tracking branch 'origin/main' into numpy_1.25

93ac652

Add mixed signedness ufuncs to CUDA

55855d7

As titled.

stuartarchibald reviewed Jul 12, 2023

View reviewed changes

numba/cuda/ufuncs.py Outdated Show resolved Hide resolved

stuartarchibald reviewed Jul 12, 2023

View reviewed changes

esc unassigned gmarkall, stuartarchibald and guilhermeleobas Jul 13, 2023

esc requested a review from guilhermeleobas July 13, 2023 14:21

Change array overloads for NumPy 1.25

8f47d8c

As requested in review.

gmarkall reviewed Jul 26, 2023

View reviewed changes

stuartarchibald mentioned this pull request Jul 26, 2023

Updates the minimum supported NumPy to 1.22. #9093

Closed

Test signed / unsigned comparisons in basic ufunc tests

989e81d

Merge branch 'main' into numpy_1.25

eb74738

Use Python 3.10 for gpuCI

3f46ed1

NumPy 1.25 packages are not provided for Python < 3.10 on the default conda channel, so we need to use a newer version.

gpuCI: Use Python 3.10 environment in 3.8 image

443c2ff

There isn't a suitable RAPIDS image with 3.10 that keeps the versions of all other components the same, so instead we'll keep the 3.8 image and create a 3.10 environment in it.

Use round_

0c086cf

As titled.

apmasell closed this Jul 28, 2023

gmarkall mentioned this pull request Jul 28, 2023

NumPy 1.25 support (PR #9011) continued #9105

Merged

esc added a commit that referenced this pull request Aug 1, 2023

Merge pull request #9105 from gmarkall/numpy_1.25

19a0196

NumPy 1.25 support (PR #9011) continued

bdice mentioned this pull request Sep 21, 2023

[BUG] Pin wheel builds to numpy <1.25 rapidsai/cudf#14155

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Implementation and test updates to support NumPy 1.25 #9011

Implementation and test updates to support NumPy 1.25 #9011

apmasell commented Jun 12, 2023

stuartarchibald commented Jun 13, 2023

gmarkall Jun 27, 2023

apmasell Jun 27, 2023

gmarkall Jun 27, 2023

gmarkall Jun 27, 2023

gmarkall Jun 27, 2023

gmarkall Jun 27, 2023

apmasell Jun 27, 2023

gmarkall Jun 27, 2023

gmarkall left a comment

gmarkall Jun 28, 2023

apmasell Jun 28, 2023

seberg commented Jun 28, 2023

stuartarchibald commented Jul 11, 2023

stuartarchibald Jul 12, 2023

sklam Jul 12, 2023

apmasell Jul 13, 2023

stuartarchibald Jul 27, 2023

gmarkall Jul 26, 2023

apmasell Jul 26, 2023

gmarkall Jul 27, 2023

apmasell Jul 27, 2023

gmarkall commented Jul 27, 2023

gmarkall commented Jul 27, 2023

gmarkall commented Jul 27, 2023

gmarkall commented Jul 27, 2023

gmarkall commented Jul 27, 2023

gmarkall commented Jul 27, 2023

apmasell commented Jul 27, 2023

		@@ -30,7 +30,7 @@ conda list
		# Use conda-forge for NumPy 1.24 - at the time of writing it is not available

	# Use conda-forge for NumPy 1.24 - at the time of writing it is not available
	# Use conda-forge for NumPy 1.25 - at the time of writing it is not available

		@@ -16,7 +16,7 @@ set PIP_INSTALL=pip install -q

		@rem Use conda-forge for NumPy 1.24 - at the time of writing it is not available

	@rem Use conda-forge for NumPy 1.24 - at the time of writing it is not available
	@rem Use conda-forge for NumPy 1.25 - at the time of writing it is not available

	if %NUMPY% GEQ 1.24 (set NUMPY_CHANNEL_PKG="conda-forge::numpy") else (set NUMPY_CHANNEL_PKG="numpy")
	if %NUMPY%==1.25 (set NUMPY_CHANNEL_PKG="conda-forge::numpy") else (set NUMPY_CHANNEL_PKG="numpy")

		@@ -361,6 +361,38 @@ def int_ne_impl(context, builder, sig, args):
		return impl_ret_untracked(context, builder, sig.return_type, res)


		def int_su_cmp(op):

Implementation and test updates to support NumPy 1.25 #9011

Implementation and test updates to support NumPy 1.25 #9011

Conversation

apmasell commented Jun 12, 2023

stuartarchibald commented Jun 13, 2023

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

gmarkall left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

seberg commented Jun 28, 2023

stuartarchibald commented Jul 11, 2023

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

gmarkall commented Jul 27, 2023

gmarkall commented Jul 27, 2023

gmarkall commented Jul 27, 2023

gmarkall commented Jul 27, 2023

gmarkall commented Jul 27, 2023

gmarkall commented Jul 27, 2023

apmasell commented Jul 27, 2023