Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Building wheels with GMP et al #352

Open
oscarbenjamin opened this issue Dec 20, 2022 · 48 comments
Open

Building wheels with GMP et al #352

oscarbenjamin opened this issue Dec 20, 2022 · 48 comments

Comments

@oscarbenjamin
Copy link

If this is a bad place to open this discussion then let me know and I'll take it elsewhere.

I've been working on python-flint which is another project that packages similar libraries to gmpy. Along with GMP and MPFR, python-flint also includes Flint and Arb so it is similar in some ways to gmpy but with a different overall scope. The main thing I've been working on is producing wheels (flintlib/python-flint#1) and I'm pretty close to having that working with some of that work inspired by the way that gmpy does things.

I wonder if there is scope for collaborating and learning from each other about exactly what is a good way to build something like this because it seems like a very much uphill struggle with current Python packaging infrastructure (although things are getting better over time).

In particular I have questions right now about how exactly gmpy makes wheels for Apple arm64. I see that recent commits include a patch to GMP for this platform. Is that necessary? Does something not work without it?

Has the gmpy project considered moving to meson or cmake etc because I see some suggestion that other scientific Python projects are moving that way. Could there be any scope to work together on moving towards new build systems?
@casevh
Copy link
Collaborator

casevh commented Dec 21, 2022

Hi Oscar,

This is a fine forum for these discussions.

I skimmed your work that you've done on python-flint. I think you've encountered most of the same issues with complex builds.

Regarding Apple arm64 builds - yes, the patch is required. Apple reserved a register for their use. GMP 6.2.1 uses that register. The patch modifies GMP 6.2.1 to save/restore that register. (https://gmplib.org/list-archives/gmp-discuss/2022-July/006821.html) Without the patch, it will crash frequently. (See #350)

Some comments on Windows

Compiling gmpy for Windows has always been a challenge. -cmingw32 worked well for 32-bit versions of Python but required extensive hacking to use mingw64 compilers on 64-bit Windows. (I originally used MPIR and MSVC but MSVC didn't support --enable-fat. I didn't want to compile for the lowest common CPU type nor try to release CPU specific versions.)

I compile the Windows binaries locally. I just extracted a copy of the cygwincompiler.py file from numpy and replace the file that comes with Python with a slightly modified copy. I don't compile anything else on Windows so it works for me. It would be great if the numpy version of -cmingw32 could be added to setuptools as -cmingw64. That would eliminate any risk of breakage for existing code that patches -cmingw32 on the fly.

I know it is possible to compile Cython extensions on Windows using MSVC and link to the GMP/MPFR/MPC dlls. (See #320) At some point, I'd like to compile GMP/MPFR/MPC with a version of mingw64 that links to Microsoft's ucrt and then compile gmpy with MSVC. The ucrt dlls could be release as "wingmpy2-lib" package and become a basis for other projects.

Regarding meson or cmake, I'm open to anything that it easier and more predictable, especially over the long run.

Regards,
Case

@oscarbenjamin
Copy link
Author

Without the patch, it will crash frequently. (See #350)

Thanks. That's what I was wondering. It seems this topic comes up monthly on the GMP mailing list. I wonder when there might be a release with this patch.

I don't compile anything else on Windows so it works for me. It would be great if the numpy version of -cmingw32 could be added to setuptools as -cmingw64.

I guess that's implicitly what we're depending on by using numpy.distutils in python-flint. I suspect that adding this to setuptools is not going to happen but I might be wrong. AFAICT Core Python and PyPA etc basically expect that MSVC will be used on Windows. That doesn't work for many key scientific packages (e.g. scipy) but it seems that their fix is to move away from setuptools to things like meson. I might be wrong but it seems like setuptools doesn't have resource to support non-MSVC compilers and the packages that do want other compilers are moving away from setuptools.
https://labs.quansight.org/blog/2021/07/moving-scipy-to-meson
pypa/setuptools#2372

The ucrt dlls could be release as "wingmpy2-lib" package and become a basis for other projects.

It would be a lot better if we could depend on something like that. How exactly would it work for building a downstream project that depended on these DLLs at the C level? I'd still need all headers etc to build python-flint but can that all be packaged up so that python-flint CI just installs the wingmpy2-lib wheel and then uses plain setuptools/MSVC to build a Cython extension based on that?

That would be good although I'd still need to then build Flint and Arb so I'm not sure how much it would simplify anything for making python-flint wheels... I guess not duplicating the DLLs on disk or in memory is also a good thing and not having to carry patches for GMP etc. So yes if it just works(!) then it would simplify things.

Then again it would be better if it was just easier to build stuff on Windows using either mingw64 or MSVC and have everything else taken care of. Ideally I'd just say in the cibuildwheel config what sort of compiler I'd want to use and everything else would be figured out by the build tools without me needing to manually code everything. It should be possible for both C projects and C-based Python extension modules to specify their build in a sufficiently platform independent way that which compiler is used is just a configuration variable or something but it definitely doesn't feel that simple right now...

Since you mention ucrt the python-flint wheels currently link against msvcrt.dll. They seem to work fine but I assume that's potentially problematic and it would be better to link against ucrt. I'm sure I saw somewhere that msys2's ucrt support was somehow experimental or not production ready or something but I can't find that now although there's some information about ucrt here:
https://www.msys2.org/docs/environments/
If I understand correctly it should be straight-forward to switch to ucrt by just changing this line in CI:
https://github.com/fredrik-johansson/python-flint/blob/e08d3dfc236f6b38fcc11bcde5c621993761a806/.github/workflows/buildwheel.yml#L24
Maybe I should test that...

More generally though it seems like you really need to be an expert in packaging to make any of this work and it's really not my domain of expertise. I know of this project to help with these general problems:
https://discuss.python.org/t/modern-way-to-build-packages-with-c-cython-extensions/15050/28
https://iscinumpy.dev/post/scikit-build-proposal/
Perhaps @henryiii do you have any advice about what might be a good path forwards for projects like these?

@oscarbenjamin
Copy link
Author

I'm sure I saw somewhere that msys2's ucrt support was somehow experimental

It's here that I see the note about it being experimental but I'm not sure what exactly it means:
https://github.com/marketplace/actions/setup-msys2#build-matrix

@rgommers
Copy link

rgommers commented Jan 4, 2023

I guess that's implicitly what we're depending on by using numpy.distutils in python-flint. I suspect that adding this to setuptools is not going to happen but I might be wrong

AFAIK the setuptools maintainers are still open to any contributions that folks need, including any features that numpy.distutils has. That said, I don't think anyone is working on that and moving to another build system is probably easier than starting to add things you need to setuptools.

@oscarbenjamin
Copy link
Author

Thanks Ralf.

I don't particularly want to become a maintainer of these parts of setuptools so if no one else is interested working on it then I'd rather switch towards tools that other projects are going to use and that people with better expertise than me are going to maintain.

This was referenced Jan 8, 2023
Merged
Closed
@oscarbenjamin oscarbenjamin mentioned this issue Aug 14, 2023
Closed
@oscarbenjamin
Copy link
Author

I have opened a more recent python-flint issue to discuss the Python 3.12 problem:
flintlib/python-flint#52

@casevh What are the current plans for gmpy2 in Python 3.12?

@casevh
Copy link
Collaborator

casevh commented Aug 16, 2023

What are the current plans for gmpy2 in Python 3.12?

My current approach relies on hacking the local copies of setuptools by overwriting cygwincompiler.py. It is not a realistic option for anyone else to use.

The following would be nice but it's probably too late to be viable - add a new compiler type called mingw64.

  • This would add the numpy.distutils support for the modern MinGW64 compilers.
  • The changes to distutils are simple. There are no changes to the existing mingw32 compiler environment. Any of the existing hacks to use mingw32 will continue to work.
  • Existing projects will need to opt-in by specifying the new compiler.

@rgommers Would such a change be acceptable for consideration into setuptools? (I won't be offended if the answer is no.)

@oscarbenjamin I will add comments to flintlib/python-flint#52 regarding GMP builds.

@rgommers
Copy link

@rgommers Would such a change be acceptable for consideration into setuptools? (I won't be offended if the answer is no.)

I'm not a setuptools maintainer, but my understand is that yes, they're open to such things. And I agree that adding support for a new compiler is fairly easy to do and non-intrusive. So it's probably a single PR with quite low associated maintenance costs after that. You'd have to make that PR though. setuptools releases quite frequently, so this can be available pretty quickly.

@casevh
Copy link
Collaborator

casevh commented Sep 11, 2023

Current update:

  • I was able to compile GMP/MPFR/MPC and link them to ucrt. Compiling gmpy2 was successful but the test suite crashed.
  • I am repeat the build process with but using mscrt.

More details to follow.

@oscarbenjamin
Copy link
Author

  • I was able to compile GMP/MPFR/MPC and link them to ucrt. Compiling gmpy2 was successful but the test suite crashed.

This sounds like my epxperience in:
flintlib/python-flint#41

@casevh
Copy link
Collaborator

casevh commented Oct 13, 2023

I've added the GMP, MPFR, and MPC DLLs and library files to the gmpy2 repository. Look in gmpy\mingw64\winlibs. Basic tests pass and I need to do some cleanup yet.

But no hacks or modifications are needed to compile with MSVC and link to those DLLs.

@oscarbenjamin
Copy link
Author

Is this line correct:

gmpy/mingw64/msys2_build.txt

Line 105 in 8c58be9

py -3.12 setup.py build_ext -cmingw32 -Imingw64\gmpy2 -Lmingw64\gmpy2 -f bdist_wheel

It still has -cmingw32...

@casevh
Copy link
Collaborator

casevh commented Oct 14, 2023 via email

@oscarbenjamin
Copy link
Author

I've added the GMP, MPFR, and MPC DLLs and library files to the gmpy2 repository.

Are these all linked with ucrt? I think this means that they are linked with msvcrt:

gmpy/mingw64/msys2_build.txt

Line 18 in 8c58be9

2. Reboot and launch the "MSYS2 MINGW64" terminal window. Upgrade the user

@casevh
Copy link
Collaborator

casevh commented Oct 14, 2023 via email

@casevh
Copy link
Collaborator

casevh commented Oct 15, 2023 via email

@dimpase
Copy link
Contributor

dimpase commented Dec 1, 2023

It's a bit off-topic, but could python-flint use gmpy2 rather than their own copy of libgmp? One advantage is a smaller foothold for dynamic libraries.

@oscarbenjamin
Copy link
Author

If you build things locally then there is no reason why python-flint and gmpy2 cannot share the same libgmp. That is also what would happen with conda or with e.g. Linux distros (if they start shipping python-flint).

It is just for PyPI wheels that gmpy2 and python-flint need to bundle libgmp. There is not currently a clear way to have an ABI dependency between wheels. Basically what you want is to be able to say "this exact python-flint wheel depends on that exact gmpy2 wheel" even though the source compatibility constraints are much looser than that. There is currently no way to express that dependency relationship though.

For python-flint this does not save much in terms of disk space because libflint is about 100x larger than libgmp anyway. Also mpfr is a shared dependency but this is still much smaller than libflint. In my local build of python-flint I have a 69MB lib directory of which 64MB is libflint.so.

Another potential benefit of having python-flint depend on gmpy2 is to enable conversion between e.g. gmpy2.mpz and flint.fmpz but again that would need ABI compatibility for wheels.

@dimpase
Copy link
Contributor

dimpase commented Dec 4, 2023

Note that at present any package which needs cython-level interface to gmpy2 cannot simply use its binary wheel, as they lack GMP headers, and compilation will fail. Shouldn't gmpy2 also ship gmp.h and all the other library headers used?

Well, it's probably a general deficiency of binary wheels, not only ones here.

@casevh
Copy link
Collaborator

casevh commented Dec 4, 2023 via email

@dimpase dimpase mentioned this issue Dec 4, 2023
Merged
5 tasks
@skirpichev
Copy link
Contributor

@dimpase, see #447

@casevh, at least mentioned issue should be solved.

@casevh
Copy link
Collaborator

casevh commented Dec 5, 2023

Here are quick instructions on how to build a gmpy2 binary wheel for Windows.

The commands are run from the "x64 Native Tools Command Prompt"

  1. git clone https://github.com/aleaxit/gmpy.git
  2. cd gmpy
  3. py -3.10 -m pip install --upgrade pip setuptools wheel pytest hypothesis build cython
  4. py -3.10 -m build --wheel
  5. py -3.10 -m pip install dist\gmpy2-2.2.0a2-cp310-cp310-win_amd64.whl --force-reinstall

The headers and DLLs are installed in the same location as gmpy2.h (site-packages\gmpy2\gmpy2). A Cython test program is in gmpy\test_cython. It should run with py -3.10 runtests.py but it fails for me.

GMP, MPFR, and MPC were compiled with MinGW64 and linked against UCRT.

The only change made to GMP is mp_bitcnt_t is now an unsigned long long.

Any help from this point is appreciated.

@skirpichev
Copy link
Contributor

It should run with py -3.10 runtests.py but it fails for me.

Here is how it works in our CI workflow:

gmpy/.github/workflows/pip_install_gmpy2.yml

Lines 104 to 107 in 831782a

- run: pip install --upgrade pip
- run: pip --verbose install -e .[tests]
- run: pytest test/
- run: PYTHONPATH=`pwd`/gmpy2 python test_cython/runtests.py

As you can see, cython tests are working.

Unfortunately, I don't know yet how to adapt this approach for cibuildwheel builds (i.e. change compiler from MSVC).

@skirpichev skirpichev mentioned this issue Dec 20, 2023
Closed
@skirpichev skirpichev mentioned this issue Feb 9, 2024
Closed
@casevh
Copy link
Collaborator

casevh commented Feb 12, 2024

Update on gmpy2 binaries for Windows

I currently compile GMP, MPFR, and MPC using MSYS2 with a version of MinGW64 linked to ucrt. The resulting header, library, and DLLs are added to the git repository. The actual gmpy2 binaries can be compiled by MSVC with just a git checkout and then "py -3.12 -m build --wheel". When the resulting wheel is installed, Cython and C extensions can use the DLLs bundled with binary wheel.

Challenges

Cython and C-API extentions should use the same libraries as gmpy2. To avoid naming collisions, I want to rename DLLs. I've done this but it requires the renaming must be done before compiling gmpy2. I'm locally using libgmpy2_2_2_gmp-10.dll instead of libgmp-10.dll.

Are there any concerns with predictable name-mangling of the GMP, MPFR, and MPC DLLs? (Note: the header and lib file names do not change.)

Including Windows specfic DLLs in a source distribution is not appropriate. So compiling gmpy2 from source would need to trigger a MSYS2/MingGW64 phase and then a DOS/MSVC phase to create the actual binary wheels. I don't know how to automate a build workflow that requires both MSYS2/MinGW64 and DOS/MSVC.

Are there any examples of such a workflow?

The current state is that it is easy to create binary wheels for Windows as long as you start with a git checkout and have MSVC. That is a significant improvement over prior versions. I consider it good-enough.

I've committed the latest name-mangled version for testing.

@oscarbenjamin
Copy link
Author

Are there any concerns with predictable name-mangling of the GMP, MPFR, and MPC DLLs?

Why not use delvewheel for this?

More generally it would be good to use cibuildwheel and have all of this running in CI.

I don't know how to automate a build workflow that requires both MSYS2/MinGW64 and DOS/MSVC.

Are there any examples of such a workflow?

Neither do I unfortunately. I think though that unless you pass compiler=mingw32 or similar then if MSVC is available it will be used by setuptools automatically. For python-flint there is an awkward workaround to prevent MSVC from being used:
https://github.com/flintlib/python-flint/blob/30e71dcd2b730c4d5ea4ecf692e547b70a0477ca/bin/cibw_before_all_windows.sh#L8-L14

@casevh
Copy link
Collaborator

casevh commented Feb 12, 2024

Are there any concerns with predictable name-mangling of the GMP, MPFR, and MPC DLLs?

Why not use delvewheel for this?

I'm trying to support extentions that use gmpy2's C-API. The fundamental issue is that GMP allows its memory allocation functions to be changed. This is a global change that is inherited by MPFR and MPC. Any code that relies on gmpy2's C-API must use the same memory management functions. I think this is best done by having those extensions use the same DLLs and I think that is best done with consistent, predictable names for the DLLs.

The C-API used to work well on *NIX platforms because gmpy2 and extensions would generally use the system provided GMP, MPFR, and MPC libraries. Modern development environments and the proliferation of binary wheels make this a difficult problem.

Since Windows doesn't provide standardized versions of GMP, MPFR, and MPC, I'm trying to use gmpy2's versions as the "system" libraries for extensions that use gmpy2's C-API. And it really works well. If you have gmpy2 installed, compiling C/Cython extensions is really quick and easy and only relies on the standard MSVC toolchain.

More generally it would be good to use cibuildwheel and have all of this running in CI.

I don't know how to automate a build workflow that requires both MSYS2/MinGW64 and DOS/MSVC.
Are there any examples of such a workflow?

Neither do I unfortunately. I think though that unless you pass compiler=mingw32 or similar then if MSVC is available it will be used by setuptools automatically. For python-flint there is an awkward workaround to prevent MSVC from being used: https://github.com/flintlib/python-flint/blob/30e71dcd2b730c4d5ea4ecf692e547b70a0477ca/bin/cibw_before_all_windows.sh#L8-L14

The MSYS2/MinGW64 toolchain is only needed to create the GMP, MPFR, and MPC libraries. The actual gmpy2 build only uses MSVC. A solution for running the cibw_before.... script in MSYS2/MinGW64 and the gmpy2 build in DOS/MSVC would be perfect. No workarounds are required.

@dimpase
Copy link
Contributor

dimpase commented Feb 12, 2024

maybe it's time to create pip-installable wheels for gmp,mpfr, mpc? just like what scipy project has done for openblas (which is in a similar league to gmp, but for numeric linear algebra)

@oscarbenjamin
Copy link
Author

If you have gmpy2 installed, compiling C/Cython extensions is really quick and easy and only relies on the standard MSVC toolchain.

I'm wondering if this approach could work for python-flint. We would still need to build libflint though and I am not sure if that could be done with MSVC while still being compatible with the MinGW builds of GMP and MPFR.

If it meant that python-flint could avoid needing to use MinGW then it might be worth doing.

A solution for running the cibw_before.... script in MSYS2/MinGW64 and the gmpy2 build in DOS/MSVC would be perfect. No workarounds are required.

You can see how this is done here:
https://github.com/flintlib/python-flint/blob/30e71dcd2b730c4d5ea4ecf692e547b70a0477ca/.github/workflows/buildwheel.yml#L22-L39
The msys2 action provides an msys2 command on PATH that can be used to run commands in the msys2 shell. Anything not run in the msys2 shell (including cibuildwheel in this case) would see the normal DOS/MSVC paths.

@casevh
Copy link
Collaborator

casevh commented Feb 12, 2024

I'm wondering if this approach could work for python-flint. We would still need to build libflint though and I am not sure if that could be done with MSVC while still being compatible with the MinGW builds of GMP and MPFR.

If it meant that python-flint could avoid needing to use MinGW then it might be worth doing.

The GMP, MPFR, and MPC libraries are linked to Microsoft's ucrt.

If you are creating a Windows binary wheel of gmpy2, there is a pre-work phase which requires MSYS2 and a version of MinGW64 that links to ucrt, and some copying of files to appropriate locations (DOS). Then the actual compilation of gmpy2 just uses MSVC. I've not had any issue with gmpy2.

To compile a C-API extension, all that is required is use of a Python interpreter that has gmpy2 installed. Just need to use that installation of Python for compiling the extension. See demo/setup.py & demo/gmpy2_demo.c and test_cython for examples.

A solution for running the cibw_before.... script in MSYS2/MinGW64 and the gmpy2 build in DOS/MSVC would be perfect. No workarounds are required.

You can see how this is done here: https://github.com/flintlib/python-flint/blob/30e71dcd2b730c4d5ea4ecf692e547b70a0477ca/.github/workflows/buildwheel.yml#L22-L39 The msys2 action provides an msys2 command on PATH that can be used to run commands in the msys2 shell. Anything not run in the msys2 shell (including cibuildwheel in this case) would see the normal DOS/MSVC paths.

That sounds perfect. I'll look at it.

@oscarbenjamin
Copy link
Author

If you are creating a Windows binary wheel of gmpy2, there is a pre-work phase which requires MSYS2 and a version of MinGW64 that links to ucrt, and some copying of files to appropriate locations (DOS). Then the actual compilation of gmpy2 just uses MSVC. I've not had any issue with gmpy2.

In the case of python-flint we also need a prework phase to build GMP, MPFR and FLINT before building the python-flint extension modules. Leveraging gmpy2's C API we could get GMP and MPFR for free but would then still need to build FLINT which as a C library depends on both GMP and MPFR. I am wondering about whether we could build FLINT with MSVC.

The only example I know of that builds FLINT with MSVC is this one which uses GMP, MPFR and pthreads from vcpkg:
https://github.com/flintlib/flint/blob/1cbdc22e70897b3c59748f56cf47932e4045471d/.github/workflows/CI.yml#L528-L549

I am not sure how I could do the same to build FLINT against GMP and MPFR from gmpy2 using MSVC.

Maybe this is not difficult but I don't have much experience with MSVC, vcpkg, cmake, ...

Somehow we need to pass some information from gmpy2 to say where the headers and DLLs are for GMP and MPFR and then when libflint.dll and the flint headers are built we need to have those somewhere that MSVC would find when the extension modules are being built.

If we still need MinGW to build FLINT then there is not much benefit in trying to use the GMP and MPFR that are provided by gmpy2 because it's easy to build them with MinGW as well, they don't add much to build-time or wheel-size etc. There would be some benefit but also the downside of the ABI coupling between gmpy2 and python-flint wheels.

@keykeykeykeyk
Copy link

keykeykeykeyk commented Feb 21, 2024 •
edited
Loading

Here are quick instructions on how to build a gmpy2 binary wheel for Windows.

The commands are run from the "x64 Native Tools Command Prompt"

  1. git clone https://github.com/aleaxit/gmpy.git
  2. cd gmpy
  3. py -3.10 -m pip install --upgrade pip setuptools wheel pytest hypothesis build cython
  4. py -3.10 -m build --wheel
  5. py -3.10 -m pip install dist\gmpy2-2.2.0a2-cp310-cp310-win_amd64.whl --force-reinstall

The headers and DLLs are installed in the same location as gmpy2.h (site-packages\gmpy2\gmpy2). A Cython test program is in gmpy\test_cython. It should run with py -3.10 runtests.py but it fails for me.

I got

File "C:\...\Python311\Lib\site-packages\gmpy2\__init__.py", line 1, in <module>
    from .gmpy2 import *
ImportError: DLL load failed while importing gmpy2

is this the same as your error? To check i dont make something else wrong.

@casevh
Copy link
Collaborator

casevh commented Feb 21, 2024

I spun up a new VM and recreated the issue. Will investigate.

@casevh
Copy link
Collaborator

casevh commented Feb 27, 2024 •
edited
Loading

@keykeykeykeyk

Are the contents of site\packages\gmpy2\gmpy2\gmpy2.h valid? (gmpy2.h is a symbolic link which doesn't always work as expected on Windows.)

@keykeykeykeyk
Copy link

Are the contents of site\packages\gmpy2\gmpy2\gmpy2.h valid? (gmpy2.h is a symbolic link which doesn't always work as expected on Windows.)

I get site-packages\gmpy2\gmpy\gmpy2\gmpy2.h as i start under site-packages\gmpy2
After i install the wheel, there is also a gmpy2.h under site-packages\gmpy2
However the \src directory is under site-packages\gmpy2\gmpy. seems like a path issue?

@dimpase
Copy link
Contributor

dimpase commented Feb 27, 2024

IMHO on Windows it's pretty much hopeless to get symbolic links working properly. Better just copy files.

@casevh
Copy link
Collaborator

casevh commented Feb 29, 2024 via email

@skirpichev
Copy link
Contributor

dll2lib.bat is still mentioned in msys2_build.txt

@casevh
Copy link
Collaborator

casevh commented Feb 29, 2024

dll2lib.bat isn't used for name-mangling and is still needed to create the required lib files. I used a different utility called rename-dll (sp?) for name-mangling.

@skirpichev
Copy link
Contributor

dll2lib.bat isn't used for name-mangling and is still needed to create the required lib files.

Yes. And it's missing from in sources.

I used a different utility called rename-dll (sp?) for name-mangling.

Why not delvewheel? It seems to be working for flint people.

@casevh
Copy link
Collaborator

casevh commented Mar 1, 2024 •
edited
Loading

I have a fundamental conflict with name-mangling to random names and supporting the C-API. I will only support the C-API if the same DLL names for GMP, MPFR, and MPC are used for all offical Windows binary wheels of gmpy2 2.2.x. There is no other way to guarantee that extensions that use gmpy2's C-API will work consistently.

I could remove the C-API on Windows but I actually want to use it.

I would like to disable the C-API for all binary wheels since there is no way to gurarantee that an extension will use the same DLLs as gmpy2. But I don't know if that is possible.

It's not an issue of name-mangling, the key issue is that the GMP, MPFR, and MPC names can change for every release. So extensions would need to be re-compiled for every different release of a binary wheel.

@skirpichev
Copy link
Contributor

I did attempts to use bundled winlibs/ files to built win wheels in CI (this branch: https://github.com/skirpichev/gmpy/tree/win-wheels), but without success: https://github.com/skirpichev/gmpy/actions/runs/8129156379/job/22215901783 (Maybe I should try different win version?)

@oscarbenjamin
Copy link
Author

but without success:

It doesn't look GMP was built. Should that not be done in CIBW_BEFORE_ALL?

@skirpichev
Copy link
Contributor

skirpichev commented Mar 4, 2024 •
edited
Loading

It doesn't look GMP was built.

Source tree has built libraries. The idea was first try to use this.
Edit: Oh, I see - DLL's were removed in recent commits. I tried to restore this, but this also doesn't work.

@oscarbenjamin
Copy link
Author

It is better if everything is built in CI. If mingw is used to build GMP etc then the process should not be much different from what python-flint does. Just there are some steps needed to process the DLLs which would need to be different. In the CIBW_BEFORE_BUILD_WINDOWS step python-flint does this:
https://github.com/flintlib/python-flint/blob/1ce152dffc356af69b1d4c2ea0eb08854f3d733b/bin/cibw_before_build_windows.sh#L31-L37
That code for python-flint is messing with python DLLs so that mingw can build the extension module. Instead with the approach described above for gmpy2 that should probably be replaced by some code in CIBW_BEFORE_ALL_WINDOWS. It should do something with the GMP etc DLLs/headers so that MSVC can find them when building the gmpy2 extension module.

@skirpichev
Copy link
Contributor

If mingw is used to build GMP etc then the process should not be much different from what python-flint does.

I tried this (https://github.com/skirpichev/gmpy/tree/win-wheels2), following the python-flint build scripts. Perhaps, I miss something (got cannot find -lvcruntime140 on pip wheel step): https://github.com/skirpichev/gmpy/actions/runs/8151061981) obvious...

It should do something with the GMP etc DLLs/headers so that MSVC can find them when building the gmpy2 extension module.

I'll try to follow mingw64/msys2_build.txt, if above variant fails.

@casevh
Copy link
Collaborator

casevh commented Mar 5, 2024

Some comments.

Please follow the instructions in mingw64/msys2_build.txt precisely.

I use the Microsoft developer VMs ( https://developer.microsoft.com/en-us/windows/downloads/virtual-machines/ ) to create the wheels for Windows.

It would be much easier if there was support for OS specific source distributions. But I doubt that will happen.

My apologies for delays in responsding, I have too many demands on my time. I would like to focus on releasing gmpy2 2.2.0. The gmpy2 code is in a good state (Thanks Sergey!!) What is required to make the formal release?

@skirpichev
Copy link
Contributor

What is required to make the formal release?

Are we run out of time? Maybe some projects (e.g. SymPy) want this due to their release plans? If not, I think that the current issue is most important one: how to automate building and testing binary wheels for M$ Windows in CI.

I would like to see some alpha release with support for CPython 3.13. Probably, we should also document how to build cython extensions with provided macos/linux binary wheels (this is only tested on CI in #447). Maybe we should also update changelog (see #329), i.e. move all that to a single place, preferably to the sphinx docs. (I would like also get rid of using private CPython API, see #467: there is some time, before CPython 3.13 beta...)

@skirpichev skirpichev mentioned this issue Mar 7, 2024
Closed
@skirpichev
Copy link
Contributor

Please follow the instructions in mingw64/msys2_build.txt precisely.

It seems the error on my side was simple: I forgot to blacklist win32.

Here is a PR, that uses mixed toolchain: #469.
But we could build all like python-flint.

@skirpichev skirpichev mentioned this issue Mar 8, 2024
Merged
@skirpichev skirpichev mentioned this issue Apr 1, 2024
Closed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
6 participants
@rgommers @oscarbenjamin @dimpase @casevh @skirpichev @keykeykeykeyk