Building wheels for Windows

[jorisvandenbossche]

I have been looking a bit into the option to use vcpkg to get the GDAL binaries necessary for wheel building, and then combined with cibuildwheel for the wheel building automation.

Vcpkg is a "cross-platform C/C++ package manager" (using installation from source not providing binaries), and provides a recipe or "port" for GDAL: https://vcpkg.info/port/gdal (which allows you to install GDAL from source on windows with a basic vcpkg install gdal).
In Apache Arrow, we are also using it for building the wheels (and pyarrow also has a notoriously complex wheel, similar to requiring GDAL), and so in principle we could also use it for the other platforms (linux, mac) as well, and not just for windows (I just started with windows, since for the other platforms we could in principle start with mimicking fiona's wheel building infrastructure).

I suppose there are also other options for building wheels on Windows, by getting the binaries from elsewhere (eg https://www.gisinternals.com/, https://github.com/gisinternals/buildsystem/tree/refactor). But I also don't have any idea how easy that would be compared to vcpkg (which has the potential to give us a somewhat consistent build setup for all platforms).

---

Yesterday, I finally had some first success while trying out this workflow on a local windows machine. See my branch at https://github.com/jorisvandenbossche/pyogrio/tree/cibuildwheel-windows.

I managed to create a wheel, that seems to work, with the following steps:

• Install vcpkg (basically cloning their repo, see https://vcpkg.io/en/getting-started.html)
• Run vcpkg install gdal[core]:x64-windows to install the minimal gdal build
  • In addition to the basic Visual Studio Build Tools that I already had installed here, I needed to enable "Visual C++ ATL" in my install as well (to fix a "Cannot open include file: 'atlbase.h': No such file or directory" error while building gdal)
• Add some cibuildwheel configuration to pyproject.toml for building wheels on windows: see https://github.com/jorisvandenbossche/pyogrio/blob/2cb3362ce4e0ad1377223aed122ef9262c6c88c2/pyproject.toml#L10-L24 (setting up the environment variables, specifying the wheel repair command). Note that this branch also includes the changes from Allow gdal config via GDAL_INCLUDE/LIBRARY_PATH env variables (Simplify building on Windows with environment variables) #47
• In a clean python env, install pip install cibuildwheel
• Run cibuildwheel --platform windows
• Celebrate! :)

With this, I created a single wheel for Python 3.9:
pyogrio-0.3.0+18.gefdca6d.dirty-cp39-cp39-win_amd64.zip
I needed to rename it to allow it to upload to this issue, so for testing it, you can download it, rename it and install it with pip:

rename "pyogrio-0.3.0+18.gefdca6d.dirty-cp39-cp39-win_amd64.zip" "pyogrio-0.3.0+18.gefdca6d.dirty-cp39-cp39-win_amd64.whl"
pip install pyogrio-0.3.0+18.gefdca6d.dirty-cp39-cp39-win_amd64.whl

From a quick test (installing it in a clean env), this seems to work, I was able to read a zipped shapefile! (a cautious 🎉)

For someone with a Windows machine, please test this wheel

---

Next steps

What I did locally, as described above, should in principle all be doable in CI using github actions as well. So the next step is trying to do that:

• Ensure vcpkg is available/installed in github actions (it might be that is already available for windows hosts)
  • Github actions windows-2019 should include vcpkg, and there is a env variable pointing to the root path: https://github.com/actions/virtual-environments/blob/main/images/win/Windows2019-Readme.md#environment-variables
• Install gdal
  • A potential challenge here is that building it with vcpkg install gdal[core] takes quite a long time (even with the [core]), with the risk of timing out. So we will also certainly need to cache the built results, but for that of course need an initial successful build.
  • It seems that it is building both a release and debug build with vcpkg install. I didn't directly find a way to only get the release build (which should speed things up), xref How do I tell vcpkg to only build the release version on the command line microsoft/vcpkg#10683.
  • The issue seems to indicate that it is possible to set the build type to "release" when using cmake to build your package and its (vcpkg) dependencies. In this area, it might be an interesting option to explore switching from setuptools to scikit-build (which uses cmake) to build the C extensions of pyogrio itself.
• Set up the correct environment variables to find GDAL
• Run cibuildwheel in github actions (example from shapely: https://github.com/shapely/shapely/blob/main/.github/workflows/release.yml)

Help here would be very much appreciated!

[martinfleis]

Yay! It works flawlessly on my machine in a fresh conda env with geopandas-base (no other GDAL installed). Thanks a lot for looking into this! Having Windows wheels for IO would significantly ease the installation pain for many users.

[martinfleis]

A potential challenge here is that building it with vcpkg install gdal[core] takes quite a long time (even with the [core]), with the risk of timing out

How long does it take? The GHA docs say that

Each job in a workflow can run for up to 6 hours of execution time

It should not take longer that that. I think that we need to manually set higher timeout in the workflow but I guess we'll fit in 6 hours.

[jorisvandenbossche]

Yeah, 6 hours should be enough :) (on my machine, it took more than an hour)

[jorisvandenbossche]

One thing I realized is that the current wheel is missing is the "gdal data" and "proj data" (the Fiona wheel contains this in the fiona package, in addition to Fiona.libs folder with binaries).
I find it strange that this doesn't give an error, though, when reading a file (since that already accesses a CRS).

[brendan-ward]

Is it possible that we don't actually need gdal data and proj data? We are only getting or setting CRS info, but not reprojecting anything. I don't know if the gdal / proj data are used to encode / decode CRS from the underyling driver to CRS WKT.

[jorisvandenbossche]

I don't know if the gdal / proj data are used to encode / decode CRS from the underyling driver to CRS WKT.

I was assuming that it would need that, at least for eg OSRExportToWkt (since not every file format stores it as WKT, and this function interacts with PROJ). But we also only use OSRExportToWkt if there is no authority/code combo. So maybe the quick tests we did didn't run into that.

[jorisvandenbossche]

Doing a first attempt to put this in a github actions workflow -> #49 (and it only took less than an hour to build gdal core! and it gets cached after the first time).
On my fork, I already got up to running the tests, and I see several ERROR 1: PROJ: proj_identify: Cannot find proj.db logs. So I think this is an indication that we will need to add some "data path" handling to pyogrio for wheels .. (in the line of what fiona is doing at https://github.com/Toblerity/Fiona/blob/9c8fe736d4381526905c6976c399a33d10e3ecbf/fiona/_env.pyx#L269-L449)

[jorisvandenbossche]

If people want to test the latest version of the wheels, you can download the archive with wheels from https://github.com/geopandas/pyogrio/suites/5620607087/artifacts/183019252.
Unpack that, and then pip install path/to/unpacked/archive/pyogrio-....whl using the exact wheel name matching your python version.

[brendan-ward]

This has been completed for 0.4.0

[RitaAguiarAdsk]

@jorisvandenbossche The link to the archive with wheels is not working. Could you tell me the right link? I'm looking for the gdal 3.5.3 wheel for windows. Thank you.

If people want to test the latest version of the wheels, you can download the archive with wheels from https://github.com/geopandas/pyogrio/suites/5620607087/artifacts/183019252. Unpack that, and then pip install path/to/unpacked/archive/pyogrio-....whl using the exact wheel name matching your python version.

[brendan-ward]

@RitaAguiarAdsk We haven't yet released wheels with 3.5.x to pypi; sorry about that.

I just triggered a fresh build of the release workflow, which regenerated Windows wheels here: https://github.com/geopandas/pyogrio/actions/runs/3576011156