Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Remove MANIFEST.in use auto-generated one for sdists and package_data for wheels #12960

Merged
merged 4 commits into from
Mar 17, 2023

Conversation

vyasr
Copy link
Contributor

@vyasr vyasr commented Mar 16, 2023

Description

Using MANIFEST.in currently runs into a pretty nasty scikit-build bug (scikit-build/scikit-build#886) that results in any file included by the manifest being copied from the install tree back into the source tree whenever an in place build occurs after an install, overwriting any local changes. We need an alternative approach to ensure that all necessary files are included in built packages. There are two types:

  • sdists: scikit-build automatically generates a manifest during sdist generation if we don't provide one, and that manifest is reliably complete. It contains all files needed for a source build up to the cudf C++ code (which has always been true and is something we can come back to improving later if desired).
  • wheels: The autogenerated manifest is not used during wheel generation because the manifest generation hook is not invoked during wheel builds, so to include data in the wheels we must provide the package_data argument to setup. In this case we do not need to include CMake or pyx files because the result does not need to be possible to build from, it just needs pxd files for other packages to cimport if desired.

I also reverted #12945, which was a stopgap solution to avoid this underlying problem. That change would have caused import issues inside the python/cudf directory when installing (the lack of an inplace build would have made the source tree unimportable) so this fix removes that minor limitation introduced in that PR.

Checklist

  • I am familiar with the Contributing Guidelines.
  • New or existing tests cover these changes.
  • The documentation is up to date with these changes.

@vyasr vyasr added bug Something isn't working 3 - Ready for Review Ready for review by team non-breaking Non-breaking change labels Mar 16, 2023
@vyasr vyasr requested a review from a team as a code owner March 16, 2023 20:56
@vyasr vyasr self-assigned this Mar 16, 2023
@vyasr vyasr requested review from bdice and mroeschke March 16, 2023 20:56
@vyasr vyasr changed the title Remove MANIFEST and add package_data. Remove MANIFEST.in use auto-generated one for sdists and package_data for wheels Mar 16, 2023
@github-actions github-actions bot added the Python Affects Python cuDF API. label Mar 16, 2023
@vyasr vyasr marked this pull request as draft March 16, 2023 20:57
python/cudf/setup.py Outdated Show resolved Hide resolved
@vyasr vyasr marked this pull request as ready for review March 16, 2023 21:02
Copy link
Contributor

@bdice bdice left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Please verify that this doesn't break #12945 -- then feel free to merge.

build.sh Show resolved Hide resolved
@vyasr
Copy link
Contributor Author

vyasr commented Mar 17, 2023

/merge

@rapids-bot rapids-bot bot merged commit 8fbfb4a into rapidsai:branch-23.04 Mar 17, 2023
@vyasr vyasr deleted the fix/manifest branch March 17, 2023 21:13
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
3 - Ready for Review Ready for review by team bug Something isn't working non-breaking Non-breaking change Python Affects Python cuDF API.
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants