pip build isolation triggers re-download of dependencies #9439
Comments
|
You don't include |
|
Arguably pip does not need to verify the cache entry but may use whatever version I have locally installed. I may very well build torch from source myself, create a bdist_wheel and use that. pip should respect that. My point is that pip insisting to find a matching binary distribution on some server is flawed. I tried specifying |
|
The point of build isolation is not to use what you already have installed. If you want to use what you have installed, use
That's part of the dependency resolution for the build environment. It tries to find the best version of pytorch that matches your build requirements. I'm slightly surprised that you need pytorch installed at build time - that isn't usual (unless fairscale contains C code which is using a C API exposed by pytorch). You might want to check your dependencies aren't including a build-time dependency on pytorch when a runtime dependency would be enough. Assuming no errors in your dependencies, it definitely sounds to me like you have a case complex enough that you may find |
Yes, this is the case. For example, https://github.com/facebookresearch/fairscale/blob/master/fairscale/clib/fused_adam_cuda/fused_adam_cuda.cpp
Can projects opt out of build isolation? It seems like the new build isolation feature adds substantial overhead for a certain set of projects with native features. I doubt that the majority of users installing such projects will learn about |
|
You may also consider disabling build time isolation by default for projects with native dependencies. If a project's native code get's compiled against a different version at build time than what is used at runtime, it's possible to have ABI mismatch between build and runtime which can lead to deterministic and non-deterministic failures. |
|
pip doesn't "see" whether a project has native dependencies. It only sees this package needs dependencies, fetches them and runs the package through the "build backend" (in this case, setuptools). pip doesn't directly care about how the build backend creates what pip needs to move forward (metadata / installable artifacts etc). In this case, what you want to do is more carefully control the build environment, by getting different packages from different indexes. Notably, build-isolation is currently opt-in with pip, via the pyproject.toml file. If you don't want to opt into build isolation, removing that file will mean that pip won't choose that code path. It'll also mean that it won't try to "automatically" provide the undeclared build dependencies, but that's precisely what you want here. :) |
|
Thank you for clarifying @pradyunsg. I see that fairscale uses a pyproject.toml. Would they need to specify |
pradyunsg
added
the
C: build logic
Environment
Description
pip install fairscalewill retrigger a +700MB download of torch even though torch is already installed due to build isolation.Expected behavior
Local torch installation is used.
How to Reproduce
python3.8 -m pip install torch==1.7.1+cpu -f https://download.pytorch.org/whl/torch_stable.htmlpython3.8 -m pip install fairscale --verboseOutput
The text was updated successfully, but these errors were encountered: