Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[REVIEW] Avoid building cython twice #12945

Merged
merged 2 commits into from
Mar 15, 2023

Conversation

galipremsagar
Copy link
Contributor

Description

This PR moves the build_ext step to only happen when INSTALL_TARGET=='', thus avoiding the cython build to occur twice when build.sh is invoked.

Checklist

  • I am familiar with the Contributing Guidelines.
  • New or existing tests cover these changes.
  • The documentation is up to date with these changes.

@galipremsagar galipremsagar added bug Something isn't working Python Affects Python cuDF API. 4 - Needs cuDF (Python) Reviewer non-breaking Non-breaking change labels Mar 14, 2023
@galipremsagar galipremsagar self-assigned this Mar 14, 2023
@github-actions github-actions bot removed the Python Affects Python cuDF API. label Mar 14, 2023
Copy link
Contributor

@bdice bdice left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This looks good to me -- but I feel like there was some context about why this works as it does. @vyasr Was there some issue where scikit-build should have known to "just install" in the second command? If not, then this is probably fine to merge as-is.

build.sh Outdated Show resolved Hide resolved
Co-authored-by: Bradley Dice <[email protected]>
Copy link
Contributor

@ttnghia ttnghia left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The issue annoyed me for a long time. Thanks for this.

@galipremsagar galipremsagar added 5 - Ready to Merge Testing and reviews complete, ready to merge and removed 4 - Needs cuDF (Python) Reviewer labels Mar 14, 2023
@galipremsagar
Copy link
Contributor Author

/merge

@rapids-bot rapids-bot bot merged commit 6c8bf45 into rapidsai:branch-23.04 Mar 15, 2023
@vyasr
Copy link
Contributor

vyasr commented Mar 15, 2023

This looks good to me -- but I feel like there was some context about why this works as it does. @vyasr Was there some issue where scikit-build should have known to "just install" in the second command? If not, then this is probably fine to merge as-is.

The pair of commands predate the scikit-build transition. At the time I left it as is because I assumed that users of build.sh may originally have been relying on being able to import the package from inside the directory. The main difference between the behavior of build_ext followed by install vs just install (before the pyproject.toml transition) is that when only doing install you'll get import errors when you try to e.g. import cudf from inside the python/cudf directory because Python will prefer the local directory even though it's not a package. Running both commands makes that work as expected. However this is going to become the norm anyway when we move build.sh to use pip install commands, so I'm OK starting to impose that limitation now. Moving metadata to pyproject.toml has other subtle implications that we may as well start figuring out now as well.

vyasr added a commit to vyasr/cudf that referenced this pull request Mar 16, 2023
rapids-bot bot pushed a commit that referenced this pull request Mar 17, 2023
… for wheels (#12960)

Using MANIFEST.in currently runs into a pretty nasty scikit-build bug (scikit-build/scikit-build#886) that results in any file included by the manifest being copied from the install tree back into the source tree whenever an in place build occurs after an install, overwriting any local changes. We need an alternative approach to ensure that all necessary files are included in built packages. There are two types:
- sdists: scikit-build automatically generates a manifest during sdist generation if we don't provide one, and that manifest is reliably complete. It contains all files needed for a source build up to the cudf C++ code (which has always been true and is something we can come back to improving later if desired).
- wheels: The autogenerated manifest is not used during wheel generation because the manifest generation hook is not invoked during wheel builds, so to include data in the wheels we must provide the `package_data` argument to `setup`. In this case we do not need to include CMake or pyx files because the result does not need to be possible to build from, it just needs pxd files for other packages to cimport if desired.

I also reverted #12945, which was a stopgap solution to avoid this underlying problem. That change would have caused import issues inside the python/cudf directory when installing (the lack of an inplace build would have made the source tree unimportable) so this fix removes that minor limitation introduced in that PR.

Authors:
  - Vyas Ramasubramani (https://github.com/vyasr)

Approvers:
  - Bradley Dice (https://github.com/bdice)

URL: #12960
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
5 - Ready to Merge Testing and reviews complete, ready to merge bug Something isn't working non-breaking Non-breaking change
Projects
None yet
Development

Successfully merging this pull request may close these issues.

4 participants