Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[Python] Compatibility with NumPy 2.0 #39532

Closed
8 tasks done
jorisvandenbossche opened this issue Jan 9, 2024 · 3 comments
Closed
8 tasks done

[Python] Compatibility with NumPy 2.0 #39532

jorisvandenbossche opened this issue Jan 9, 2024 · 3 comments
Assignees
@jorisvandenbossche
Labels
Component: Python
Milestone
16.0.0

Comments

@jorisvandenbossche
Copy link
Member

jorisvandenbossche commented Jan 9, 2024 •
edited
Loading

NumPy 2.0 is coming soon, and we should ensure pyarrow is fully compatible to ensure a smooth transition of our users. Overview issue on numpy's side: numpy/numpy#24300

We already test with nightly numpy (eg https://github.com/ursacomputing/crossbow/actions/runs/7454906275/job/20283098411), and currently this build is working fine.
There is one warning about the usage of numpy.core.numerictypes that we should still solve.

Further, we need to check if we are all OK on the packaging side: do we need to pin numpy<2 for now in our upcoming release? What do we need to update in our wheel building? (stop using oldest-supported-numpy?)

cc @raulcd
@jorisvandenbossche jorisvandenbossche mentioned this issue Jan 9, 2024
Closed
@jorisvandenbossche
Copy link
Member Author

jorisvandenbossche commented Jan 9, 2024 •
edited
Loading

Some other observations / relevant bits from the linked issues:

Updating the top-post task list with some TODOs

@jorisvandenbossche jorisvandenbossche mentioned this issue Jan 9, 2024
Closed
@jorisvandenbossche jorisvandenbossche added this to the 15.0.0 milestone Jan 9, 2024
@jorisvandenbossche jorisvandenbossche added the Priority: Blocker Marks a blocker for the release label Jan 9, 2024
This was referenced Jan 9, 2024
Closed
Closed
@raulcd raulcd modified the milestones: 15.0.0, 16.0.0 Jan 11, 2024
@raulcd raulcd removed the Priority: Blocker Marks a blocker for the release label Jan 11, 2024
@jorisvandenbossche jorisvandenbossche mentioned this issue Jan 11, 2024
Merged
jorisvandenbossche added a commit that referenced this issue Feb 1, 2024
@jorisvandenbossche @pitrou
GH-39555: [Packaging][Python] Enable building pyarrow against numpy 2…
a1c1773 
….0 (#39557)

### Rationale for this change

Ensure we can build pyarrow against numpy 2.0 nightly (update pyproject.toml to allow this), and test this by building our nightly wheels with numpy nightly. This also ensures that other projects that use our nightly wheels to test together with numpy nightly can do that (numpy 2.0 changes the ABI, so to run with numpy 2.0, your package needs to be built with numpy 2.x; currently pyarrow installed with our nightly wheel will fail to import when also numpy nightly is installed).

See the parent issue #39532 for details, and https://numpy.org/devdocs/dev/depending_on_numpy.html#numpy-2-0-specific-advice for a direct link to the NumPy guidelines on updating build dependencies for NumPy 2.0.

* Closes: #39555

Lead-authored-by: Joris Van den Bossche <[email protected]>
Co-authored-by: Antoine Pitrou <[email protected]>
Signed-off-by: Joris Van den Bossche <[email protected]>
dgreiss pushed a commit to dgreiss/arrow that referenced this issue Feb 19, 2024
@jorisvandenbossche @pitrou @dgreiss
apacheGH-39555: [Packaging][Python] Enable building pyarrow against n…
9afa8ec 
…umpy 2.0 (apache#39557)

### Rationale for this change

Ensure we can build pyarrow against numpy 2.0 nightly (update pyproject.toml to allow this), and test this by building our nightly wheels with numpy nightly. This also ensures that other projects that use our nightly wheels to test together with numpy nightly can do that (numpy 2.0 changes the ABI, so to run with numpy 2.0, your package needs to be built with numpy 2.x; currently pyarrow installed with our nightly wheel will fail to import when also numpy nightly is installed).

See the parent issue apache#39532 for details, and https://numpy.org/devdocs/dev/depending_on_numpy.html#numpy-2-0-specific-advice for a direct link to the NumPy guidelines on updating build dependencies for NumPy 2.0.

* Closes: apache#39555

Lead-authored-by: Joris Van den Bossche <[email protected]>
Co-authored-by: Antoine Pitrou <[email protected]>
Signed-off-by: Joris Van den Bossche <[email protected]>
zanmato1984 pushed a commit to zanmato1984/arrow that referenced this issue Feb 28, 2024
@jorisvandenbossche @pitrou @zanmato1984
apacheGH-39555: [Packaging][Python] Enable building pyarrow against n…
9109f4e 
…umpy 2.0 (apache#39557)

### Rationale for this change

Ensure we can build pyarrow against numpy 2.0 nightly (update pyproject.toml to allow this), and test this by building our nightly wheels with numpy nightly. This also ensures that other projects that use our nightly wheels to test together with numpy nightly can do that (numpy 2.0 changes the ABI, so to run with numpy 2.0, your package needs to be built with numpy 2.x; currently pyarrow installed with our nightly wheel will fail to import when also numpy nightly is installed).

See the parent issue apache#39532 for details, and https://numpy.org/devdocs/dev/depending_on_numpy.html#numpy-2-0-specific-advice for a direct link to the NumPy guidelines on updating build dependencies for NumPy 2.0.

* Closes: apache#39555

Lead-authored-by: Joris Van den Bossche <[email protected]>
Co-authored-by: Antoine Pitrou <[email protected]>
Signed-off-by: Joris Van den Bossche <[email protected]>
@jorisvandenbossche jorisvandenbossche mentioned this issue Mar 5, 2024
Closed
thisisnic pushed a commit to thisisnic/arrow that referenced this issue Mar 8, 2024
@jorisvandenbossche @pitrou @thisisnic
apacheGH-39555: [Packaging][Python] Enable building pyarrow against n…
4ef2e5f 
…umpy 2.0 (apache#39557)

### Rationale for this change

Ensure we can build pyarrow against numpy 2.0 nightly (update pyproject.toml to allow this), and test this by building our nightly wheels with numpy nightly. This also ensures that other projects that use our nightly wheels to test together with numpy nightly can do that (numpy 2.0 changes the ABI, so to run with numpy 2.0, your package needs to be built with numpy 2.x; currently pyarrow installed with our nightly wheel will fail to import when also numpy nightly is installed).

See the parent issue apache#39532 for details, and https://numpy.org/devdocs/dev/depending_on_numpy.html#numpy-2-0-specific-advice for a direct link to the NumPy guidelines on updating build dependencies for NumPy 2.0.

* Closes: apache#39555

Lead-authored-by: Joris Van den Bossche <[email protected]>
Co-authored-by: Antoine Pitrou <[email protected]>
Signed-off-by: Joris Van den Bossche <[email protected]>
@jorisvandenbossche jorisvandenbossche mentioned this issue Mar 13, 2024
Closed
@jorisvandenbossche jorisvandenbossche added the Priority: Blocker Marks a blocker for the release label Mar 14, 2024
@rgommers rgommers mentioned this issue Apr 1, 2024
Open
jorisvandenbossche added a commit to jorisvandenbossche/arrow that referenced this issue Apr 8, 2024
@jorisvandenbossche
apacheGH-39532: [Python] Add copy keyword in Array.__array__ for nump…
2f1b8cc 
…y 2.0+ compatibility
@github-actions github-actions bot mentioned this issue Apr 8, 2024
Merged
@raulcd raulcd modified the milestones: 16.0.0, 17.0.0 Apr 8, 2024
@jorisvandenbossche jorisvandenbossche modified the milestones: 17.0.0, 16.0.0 Apr 9, 2024
@jorisvandenbossche jorisvandenbossche mentioned this issue Apr 9, 2024
Closed
jorisvandenbossche added a commit to jorisvandenbossche/arrow that referenced this issue Apr 11, 2024
@jorisvandenbossche
Merge remote-tracking branch 'upstream/main' into apachegh-39532-nump…
5b87fd3 
…y-compat-array
jorisvandenbossche added a commit to jorisvandenbossche/arrow that referenced this issue Apr 15, 2024
@jorisvandenbossche
Merge remote-tracking branch 'upstream/main' into apachegh-39532-nump…
4b630f6 
…y-compat-array
pitrou pushed a commit that referenced this issue Apr 15, 2024
@jorisvandenbossche
GH-41098: [Python] Add copy keyword in Array.__array__ for numpy 2.0+…
80258fe 
… compatibility (#41071)

### Rationale for this change

Adapting for changes in numpy 2.0 as decribed at https://numpy.org/devdocs/numpy_2_0_migration_guide.html#adapting-to-changes-in-the-copy-keyword and future changes to pass copy=True (numpy/numpy#26208)

### What changes are included in this PR?

Add a `copy=None` to the signatures of our `__array__` methods.

This does have impact on the user facing behaviour, though. Questioning that upstream at numpy/numpy#25941 (comment)

### Are these changes tested?

Yes

### Are there any user-facing changes?

No (compared to usage with numpy<2)
* GitHub Issue: #39532
* GitHub Issue: #41098

Authored-by: Joris Van den Bossche <[email protected]>
Signed-off-by: Antoine Pitrou <[email protected]>
raulcd pushed a commit that referenced this issue Apr 15, 2024
@jorisvandenbossche @raulcd
GH-41098: [Python] Add copy keyword in Array.__array__ for numpy 2.0+…
0d3af8d 
… compatibility (#41071)

### Rationale for this change

Adapting for changes in numpy 2.0 as decribed at https://numpy.org/devdocs/numpy_2_0_migration_guide.html#adapting-to-changes-in-the-copy-keyword and future changes to pass copy=True (numpy/numpy#26208)

### What changes are included in this PR?

Add a `copy=None` to the signatures of our `__array__` methods.

This does have impact on the user facing behaviour, though. Questioning that upstream at numpy/numpy#25941 (comment)

### Are these changes tested?

Yes

### Are there any user-facing changes?

No (compared to usage with numpy<2)
* GitHub Issue: #39532
* GitHub Issue: #41098

Authored-by: Joris Van den Bossche <[email protected]>
Signed-off-by: Antoine Pitrou <[email protected]>
@raulcd
Copy link
Member

raulcd commented Apr 15, 2024

@jorisvandenbossche is there something else to do on this one or are we ready for the release on this?

@jorisvandenbossche jorisvandenbossche removed the Priority: Blocker Marks a blocker for the release label Apr 15, 2024
@jorisvandenbossche
Copy link
Member Author

I think all sub-tasks are done, and this can be closed now

@jakirkham jakirkham mentioned this issue Apr 25, 2024
Closed
tolleybot pushed a commit to tmct/arrow that referenced this issue May 2, 2024
@jorisvandenbossche @tolleybot
apacheGH-41098: [Python] Add copy keyword in Array.__array__ for nump…
6034816 
…y 2.0+ compatibility (apache#41071)

### Rationale for this change

Adapting for changes in numpy 2.0 as decribed at https://numpy.org/devdocs/numpy_2_0_migration_guide.html#adapting-to-changes-in-the-copy-keyword and future changes to pass copy=True (numpy/numpy#26208)

### What changes are included in this PR?

Add a `copy=None` to the signatures of our `__array__` methods.

This does have impact on the user facing behaviour, though. Questioning that upstream at numpy/numpy#25941 (comment)

### Are these changes tested?

Yes

### Are there any user-facing changes?

No (compared to usage with numpy<2)
* GitHub Issue: apache#39532
* GitHub Issue: apache#41098

Authored-by: Joris Van den Bossche <[email protected]>
Signed-off-by: Antoine Pitrou <[email protected]>
@jakirkham jakirkham mentioned this issue May 4, 2024
Closed
vibhatha pushed a commit to vibhatha/arrow that referenced this issue May 25, 2024
@jorisvandenbossche @vibhatha
apacheGH-41098: [Python] Add copy keyword in Array.__array__ for nump…
28e281b 
…y 2.0+ compatibility (apache#41071)

### Rationale for this change

Adapting for changes in numpy 2.0 as decribed at https://numpy.org/devdocs/numpy_2_0_migration_guide.html#adapting-to-changes-in-the-copy-keyword and future changes to pass copy=True (numpy/numpy#26208)

### What changes are included in this PR?

Add a `copy=None` to the signatures of our `__array__` methods.

This does have impact on the user facing behaviour, though. Questioning that upstream at numpy/numpy#25941 (comment)

### Are these changes tested?

Yes

### Are there any user-facing changes?

No (compared to usage with numpy<2)
* GitHub Issue: apache#39532
* GitHub Issue: apache#41098

Authored-by: Joris Van den Bossche <[email protected]>
Signed-off-by: Antoine Pitrou <[email protected]>
@h-vetinari h-vetinari mentioned this issue Jun 2, 2024
Closed
@rynewang rynewang mentioned this issue Nov 16, 2024
Open
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@jorisvandenbossche jorisvandenbossche

Labels
Component: Python
Projects
None yet
Milestone
16.0.0
Development

No branches or pull requests
2 participants
@raulcd @jorisvandenbossche