Cross-platform resolving?

[AustinScola]

Description

Hi, I'm on the hunt for a Python dependency resolver that can resolve deps for different values of sys_platform and platform_machine than the machine that resolver is run from.

For example:

• From a machine with sys_platform = darwin and platform_machine = x86_64 be able to resolve deps for:
  • sys_platform = darwin, linux
  • platform_machine = x86_64, amd64

I was wondering if pipgrip is capable of this? If not could it be added? Or if you know of another resolver that can do this?

Use case / motivation / context

I would like to be able to freeze requirements that will be used on other machines that might have different OSs and chips.

Related Issues

Don't think there are any related issues?

[ddelange]

Hi @AustinScola 👋

I'm afraid this is not a trivial problem.

A good example is older versions of matplotlib which will only build from source (pip will have to execute setup.py), deciding the dependencies only on runtime based on sys etc. This could lead to incorrect trees when run from a different system.

And even if you go down that path, and you would patch pipgrip to evaluate wheels in a different environment, you'd still somehow have to patch pip itself such that pip wheel gets the appropriate wheel from PyPI (I've mocked out pip completely for tests using local wheels), a mechanic that pipgrip relies on.

Especially for different hardware architectures like arm64 (aarch64 wheels) whilst on an x86_64 machine, I think there is no way around QEMU emulation. In case pip needs to build the wheel from source, because there are missing wheels or even only an sdist available for the package (ref #40), you will need to be running (an emulation of) the actual hardware.

For the linux stuff, there are QEMU wrappers like https://github.com/marketplace/actions/run-on-architecture that will run your commands in an emulated docker environment. But in your particular case, an Intel Mac, there is no way to emulate an M1 Mac that I know of. Your best bet would probably be asking your colleagues to generate the lock...

It's all a bit of a pain really. Also building proper wheels for different architectures is something that project maintainers need to handle with care. To offload that complexity, the packaging authority came up with https://github.com/pypa/cibuildwheel, again relying on QEMU. There, the Github macos runner is used to build both x86_64 and arm64 (or universal2 for that matter) wheels for Mac from Docker. I'm not sure how they do that, because Github's macos runner is x86_64 afaik.

Hope that helps!

[AustinScola]

Darn, that is a bummer to hear this.

Thank you for the explanation.