GH-43299: [Release][Packaging] Only include pyarrow folder when finding packages on setuptools #43325

raulcd · 2024-07-18T14:52:20Z

Rationale for this change

Currently we include everything when building wheels, see:

$ pip install pyarrow
Collecting pyarrow
  Downloading pyarrow-17.0.0-cp310-cp310-manylinux_2_28_x86_64.whl (39.9 MB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 39.9/39.9 MB 33.8 MB/s eta 0:00:00
Collecting numpy>=1.16.6
  Using cached numpy-2.0.0-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (19.3 MB)
Installing collected packages: numpy, pyarrow
Successfully installed numpy-2.0.0 pyarrow-17.0.0
(test-env)  $ ls test-env/lib/python3.10/site-packages/
benchmarks/                  distutils-precedence.pth     numpy-2.0.0.dist-info/       pip-22.0.2.dist-info/        pyarrow-17.0.0.dist-info/    setuptools-59.6.0.dist-info/ 
cmake_modules/               examples/                    numpy.libs/                  pkg_resources/               scripts/                     
_distutils_hack/             numpy/                       pip/                         pyarrow/                     setuptools/

What changes are included in this PR?

Use include as seen here: https://setuptools.pypa.io/en/latest/userguide/package_discovery.html#finding-simple-packages

Are these changes tested?

Will check via the build wheel on CI

Are there any user-facing changes?

No and yes :)
We will remove unnecessary files

GitHub Issue: [Python] 17.0.0 wrongly installs multiple stray directories into site-packages #43299

raulcd · 2024-07-18T14:52:41Z

@github-actions crossbow submit -g python wheel

github-actions · 2024-07-18T14:52:47Z

⚠️ GitHub issue #43299 has been automatically assigned in GitHub to PR creator.

github-actions · 2024-07-18T14:53:50Z

Unable to match any tasks for `wheel`
The Archery job run can be found at: https://github.com/apache/arrow/actions/runs/9993787632

raulcd · 2024-07-18T14:55:55Z

@github-actions crossbow submit -g python -g wheel

raulcd · 2024-07-18T15:35:16Z

There seemed to be a minor issue:

$ ls -lrt
total 77296
drwxr-xr-x 2 raulcd raulcd     4096 jul 18 15:11 pyarrow-18.0.0.dev15.dist-info
drwxr-xr-x 2 raulcd raulcd     4096 jul 18 15:11 pyarrow.
drwxr-xr-x 9 raulcd raulcd     4096 jul 18 15:11 pyarrow
$ ls -lrt pyarrow./
total 0

raulcd · 2024-07-18T15:35:33Z

@github-actions crossbow submit wheel-manylinux-2-28-cp310-amd64

github-actions · 2024-07-18T15:37:45Z

Revision: 46d1afc

Submitted crossbow builds: ursacomputing/crossbow @ actions-db1794c125

Task	Status
wheel-manylinux-2-28-cp310-amd64

raulcd · 2024-07-18T16:00:54Z

@github-actions crossbow submit wheel-manylinux-2-28-cp310-amd64

github-actions · 2024-07-18T16:03:11Z

Revision: d954d75

Submitted crossbow builds: ursacomputing/crossbow @ actions-922921124d

Task	Status
wheel-manylinux-2-28-cp310-amd64

raulcd · 2024-07-18T16:37:13Z

@github-actions crossbow submit wheel-manylinux-2-28-cp310-amd64

github-actions · 2024-07-18T16:39:33Z

Revision: 2fa434f

Submitted crossbow builds: ursacomputing/crossbow @ actions-0cd7a8ae05

Task	Status
wheel-manylinux-2-28-cp310-amd64

raulcd · 2024-07-19T10:43:54Z

@github-actions crossbow submit wheel-manylinux-2-28-cp310-amd64

github-actions · 2024-07-19T10:46:04Z

Revision: 204a27b

Submitted crossbow builds: ursacomputing/crossbow @ actions-5d877a45d2

Task	Status
wheel-manylinux-2-28-cp310-amd64

raulcd · 2024-07-19T11:39:49Z

I am unsure on why an empty directory with pyarrow. is being added and why pyarrow.tests aren't being excluded, it will require more investigation but the initial examples, cmake_modules, benchmarks and scripts have been removed from the generated wheel:

$ ls 
pyarrow  pyarrow.  pyarrow-18.0.0.dev19-cp310-cp310-manylinux_2_28_x86_64.whl  pyarrow-18.0.0.dev19.dist-info
$ ls -lart pyarrow./
total 8
drwxr-xr-x 2 raulcd raulcd 4096 jul 19 10:59 .
drwxrwxr-x 5 raulcd raulcd 4096 jul 19 13:32 ..
$ ls pyarrow/tests/
arrow_16597.py                    interchange                 test_adhoc_memory_leak.py  test_cuda_numba_interop.py  test_exec_plan.py       test_io.py      test_scalars.py        test_udf.py
arrow_39313.py                    pandas_examples.py          test_array.py              test_cuda.py                test_extension_type.py  test_ipc.py     test_schema.py         test_util.py
arrow_7980.py                     pandas_threaded_import.py   test_builder.py            test_cython.py              test_feather.py         test_json.py    test_sparse_tensor.py  util.py
bound_function_visit_strings.pyx  parquet                     test_cffi.py               test_dataset_encryption.py  test_flight_async.py    test_jvm.py     test_strategies.py
conftest.py                       pyarrow_cython_example.pyx  test_compute.py            test_dataset.py             test_flight.py          test_memory.py  test_substrait.py
data                              read_record_batch.py        test_convert_builtin.py    test_deprecations.py        test_fs.py              test_misc.py    test_table.py
extensions.pyx                    strategies.py               test_cpp_internals.py      test_device.py              test_gandiva.py         test_orc.py     test_tensor.py
__init__.py                       test_acero.py               test_csv.py                test_dlpack.py              test_gdb.py             test_pandas.py  test_types.py

raulcd · 2024-07-19T11:40:39Z

@github-actions crossbow submit wheel-manylinux-2-28-cp310-amd64

github-actions · 2024-07-19T11:42:55Z

Revision: a1d73a2

Submitted crossbow builds: ursacomputing/crossbow @ actions-ebc75ddd3d

Task	Status
wheel-manylinux-2-28-cp310-amd64

raulcd · 2024-07-19T12:22:03Z

python/pyproject.toml

@@ -73,7 +73,9 @@ zip-safe=false
 include-package-data=true

 [tool.setuptools.packages.find]
-where = ["."]
+include = ["pyarrow"]
+exclude = ["pyarrow/tests", "pyarrow."]


this doesn't seem to have any effect, @jorisvandenbossche @pitrou any idea?
The other stray directories are filtered from the wheel now

I believe the syntax is pyarrow.tests. What does pyarrow. refer to?

there is a folder included in the wheel pyarrow. which is empty and I was trying to remove it. I'll try with pyarrow.tests

it doesn't remove tests either

We don't care about removing the tests, they are comparatively small and can serve to test PyArrow on a third-party machine.

$ du -hs venv-3.10/lib/python3.10/site-packages/pyarrow 187M venv-3.10/lib/python3.10/site-packages/pyarrow $ du -hs venv-3.10/lib/python3.10/site-packages/pyarrow/tests/ 4,0M venv-3.10/lib/python3.10/site-packages/pyarrow/tests/

what about cpp and header files, should we do something like pandas here:
https://github.com/pandas-dev/pandas/blob/main/pyproject.toml#L131

Currently the files under pyarrow/src/arrow/python are included on the wheel

I'll stress that the problem we're trying to solve is that installing a PyArrow wheel creates top-level directories outside of the pyarrow source tree. This is the urgency. Cleaning up the contents of pyarrow is a separate task, much less critical.

I guess those are negligible and we still require to distribute them on the source distitrbution:

pyarrow/src $ du -hs . 728K .

I'll stress that the problem we're trying to solve is that installing a PyArrow wheel creates top-level directories outside of the pyarrow source tree.

ok, then, this is solved by this PR, are you ok with the current changes @pitrou ?

timkpaine · 2024-07-19T16:32:06Z

was something wrong with #43281 from 3 days ago?

raulcd · 2024-07-19T16:54:40Z

was something wrong with #43281 from 3 days ago?

Sorry, I didn't see that issue, it would be good to mark the second one as a duplicate. I also think that as the issue was introduced with a wrongly configured pyproject.toml it might be worth also fixing it there

raulcd · 2024-07-22T12:47:21Z

@github-actions crossbow submit wheel-manylinux-2-28-cp310-amd64

raulcd · 2024-09-03T14:55:05Z

@github-actions crossbow submit wheel-manylinux-2-28-cp310-amd64

raulcd · 2024-09-03T14:56:13Z

@github-actions crossbow submit wheel-windows-cp39-amd64

github-actions · 2024-09-03T14:57:21Z

Revision: 915d70f

Submitted crossbow builds: ursacomputing/crossbow @ actions-5b1cb7ff99

Task	Status
wheel-manylinux-2-28-cp310-amd64

github-actions · 2024-09-03T14:58:27Z

Revision: 915d70f

Submitted crossbow builds: ursacomputing/crossbow @ actions-aca9360964

Task	Status
wheel-windows-cp39-amd64

raulcd · 2024-09-04T08:48:03Z

@github-actions crossbow submit wheel-windows-cp39-amd64

raulcd · 2024-09-04T09:44:25Z

@github-actions crossbow submit -g wheel

raulcd · 2024-09-04T10:40:45Z

@jorisvandenbossche the check is working on macOS, manylinux and Windows wheels tests now. If you can review again. Thanks!

pitrou · 2024-09-04T12:20:14Z

I think I've found the source of the stray pyarrow. directory. It's because of pypa/auditwheel#488 and the -L . option.

pitrou · 2024-09-04T12:20:39Z

@github-actions crossbow submit wheelcp312

github-actions · 2024-09-04T12:23:05Z

Revision: f87612a

Submitted crossbow builds: ursacomputing/crossbow @ actions-9544e3c33d

Task	Status
wheel-macos-monterey-cp312-amd64
wheel-macos-monterey-cp312-arm64
wheel-manylinux-2-28-cp312-amd64
wheel-manylinux-2-28-cp312-arm64
wheel-manylinux-2014-cp312-amd64
wheel-manylinux-2014-cp312-arm64
wheel-windows-cp312-amd64

pitrou · 2024-09-04T13:39:23Z

@github-actions crossbow submit -g wheel

github-actions · 2024-09-04T13:42:08Z

Revision: f87612a

Submitted crossbow builds: ursacomputing/crossbow @ actions-a951b5bfb8

Task	Status
python-sdist
wheel-macos-monterey-cp310-amd64
wheel-macos-monterey-cp310-arm64
wheel-macos-monterey-cp311-amd64
wheel-macos-monterey-cp311-arm64
wheel-macos-monterey-cp312-amd64
wheel-macos-monterey-cp312-arm64
wheel-macos-monterey-cp313-amd64
wheel-macos-monterey-cp313-arm64
wheel-macos-monterey-cp38-amd64
wheel-macos-monterey-cp38-arm64
wheel-macos-monterey-cp39-amd64
wheel-macos-monterey-cp39-arm64
wheel-manylinux-2-28-cp310-amd64
wheel-manylinux-2-28-cp310-arm64
wheel-manylinux-2-28-cp311-amd64
wheel-manylinux-2-28-cp311-arm64
wheel-manylinux-2-28-cp312-amd64
wheel-manylinux-2-28-cp312-arm64
wheel-manylinux-2-28-cp313-amd64
wheel-manylinux-2-28-cp313-arm64
wheel-manylinux-2-28-cp38-amd64
wheel-manylinux-2-28-cp38-arm64
wheel-manylinux-2-28-cp39-amd64
wheel-manylinux-2-28-cp39-arm64
wheel-manylinux-2014-cp310-amd64
wheel-manylinux-2014-cp310-arm64
wheel-manylinux-2014-cp311-amd64
wheel-manylinux-2014-cp311-arm64
wheel-manylinux-2014-cp312-amd64
wheel-manylinux-2014-cp312-arm64
wheel-manylinux-2014-cp313-amd64
wheel-manylinux-2014-cp313-arm64
wheel-manylinux-2014-cp38-amd64
wheel-manylinux-2014-cp38-arm64
wheel-manylinux-2014-cp39-amd64
wheel-manylinux-2014-cp39-arm64
wheel-windows-cp310-amd64
wheel-windows-cp311-amd64
wheel-windows-cp312-amd64
wheel-windows-cp313-amd64
wheel-windows-cp38-amd64
wheel-windows-cp39-amd64

pitrou · 2024-09-04T15:40:17Z

CI failures are unrelated. @jorisvandenbossche Do you want to take a last look?

jorisvandenbossche · 2024-09-05T07:53:23Z

I think I've found the source of the stray pyarrow. directory. It's because of pypa/auditwheel#488 and the -L . option.

Nice catch!

jorisvandenbossche · 2024-09-05T07:56:46Z

Thanks @raulcd

conbench-apache-arrow · 2024-09-05T15:05:57Z

After merging your PR, Conbench analyzed the 4 benchmarking runs that have been run so far on merge-commit f545b90.

There were no benchmark performance regressions. 🎉

The full Conbench report has more details. It also includes information about 4 possible false positives for unstable benchmarks that are known to sometimes produce them.

… finding packages on setuptools (apache#43325) ### Rationale for this change Currently we include everything when building wheels, see: ``` $ pip install pyarrow Collecting pyarrow Downloading pyarrow-17.0.0-cp310-cp310-manylinux_2_28_x86_64.whl (39.9 MB) ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 39.9/39.9 MB 33.8 MB/s eta 0:00:00 Collecting numpy>=1.16.6 Using cached numpy-2.0.0-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (19.3 MB) Installing collected packages: numpy, pyarrow Successfully installed numpy-2.0.0 pyarrow-17.0.0 (test-env) $ ls test-env/lib/python3.10/site-packages/ benchmarks/ distutils-precedence.pth numpy-2.0.0.dist-info/ pip-22.0.2.dist-info/ pyarrow-17.0.0.dist-info/ setuptools-59.6.0.dist-info/ cmake_modules/ examples/ numpy.libs/ pkg_resources/ scripts/ _distutils_hack/ numpy/ pip/ pyarrow/ setuptools/ ``` ### What changes are included in this PR? Use `include` as seen here: https://setuptools.pypa.io/en/latest/userguide/package_discovery.html#finding-simple-packages ### Are these changes tested? Will check via the build wheel on CI ### Are there any user-facing changes? No and yes :) We will remove unnecessary files * GitHub Issue: apache#43299 Lead-authored-by: Raúl Cumplido <[email protected]> Co-authored-by: Antoine Pitrou <[email protected]> Signed-off-by: Joris Van den Bossche <[email protected]>

github-actions bot added Component: Python awaiting committer review Awaiting committer review labels Jul 18, 2024

This comment was marked as outdated.

Sign in to view

raulcd changed the title ~~GH-43299: [Release][Packaging] Only include pyarrow and pyarrow.* when finding packages on setuptools~~ GH-43299: [Release][Packaging] Only include pyarrow folder when finding packages on setuptools Jul 18, 2024

raulcd marked this pull request as ready for review July 19, 2024 12:20

raulcd commented Jul 19, 2024

View reviewed changes

raulcd requested a review from anjakefala July 19, 2024 12:22

github-actions bot added awaiting changes Awaiting changes and removed awaiting committer review Awaiting committer review labels Jul 19, 2024

github-actions bot added awaiting change review Awaiting change review and removed awaiting changes Awaiting changes labels Jul 22, 2024

Temporary allow the possibility of the pyarrow. folder

915d70f

Find wheels from given path instead

52ccaf1

This comment was marked as outdated.

Sign in to view

pitrou added 2 commits September 4, 2024 14:18

Upgrade auditwheel to avoid stray libs directory

1dee776

Rename script

f87612a

jorisvandenbossche approved these changes Sep 5, 2024

View reviewed changes

github-actions bot added awaiting merge Awaiting merge and removed awaiting change review Awaiting change review labels Sep 5, 2024

jorisvandenbossche merged commit f545b90 into apache:main Sep 5, 2024
62 of 63 checks passed

jorisvandenbossche removed the awaiting merge Awaiting merge label Sep 5, 2024

jorisvandenbossche mentioned this pull request Sep 5, 2024

[Python] 17.0.0 wrongly installs multiple stray directories into site-packages #43299

Closed

GH-43299: [Release][Packaging] Only include pyarrow folder when finding packages on setuptools #43325

GH-43299: [Release][Packaging] Only include pyarrow folder when finding packages on setuptools #43325

Conversation

raulcd commented Jul 18, 2024 • edited by github-actions bot Loading

Rationale for this change

What changes are included in this PR?

Are these changes tested?

Are there any user-facing changes?

raulcd commented Jul 18, 2024

github-actions bot commented Jul 18, 2024

github-actions bot commented Jul 18, 2024

raulcd commented Jul 18, 2024

This comment was marked as outdated.

raulcd commented Jul 18, 2024

raulcd commented Jul 18, 2024

github-actions bot commented Jul 18, 2024

raulcd commented Jul 18, 2024

github-actions bot commented Jul 18, 2024

raulcd commented Jul 18, 2024

github-actions bot commented Jul 18, 2024

raulcd commented Jul 19, 2024

github-actions bot commented Jul 19, 2024

raulcd commented Jul 19, 2024

raulcd commented Jul 19, 2024

github-actions bot commented Jul 19, 2024

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

timkpaine commented Jul 19, 2024 • edited Loading

raulcd commented Jul 19, 2024

raulcd commented Jul 22, 2024

raulcd commented Sep 3, 2024

raulcd commented Sep 3, 2024

github-actions bot commented Sep 3, 2024

github-actions bot commented Sep 3, 2024

raulcd commented Sep 4, 2024

This comment was marked as outdated.

raulcd commented Sep 4, 2024

This comment was marked as outdated.

raulcd commented Sep 4, 2024

pitrou commented Sep 4, 2024

pitrou commented Sep 4, 2024

github-actions bot commented Sep 4, 2024

pitrou commented Sep 4, 2024

github-actions bot commented Sep 4, 2024

pitrou commented Sep 4, 2024

jorisvandenbossche commented Sep 5, 2024

jorisvandenbossche commented Sep 5, 2024

conbench-apache-arrow bot commented Sep 5, 2024

raulcd commented Jul 18, 2024 •

edited by github-actions bot

Loading

timkpaine commented Jul 19, 2024 •

edited

Loading