Scikit-build-core build system rework #1971
Conversation
|
This logic produces this folder structure (please ignore vector-search there): |
|
I added basic logic for downloading TileDB from release inside CMake so now we need to change CI/CD to not download release archives. |
|
I had to disable build for python 3.9, I assume that that is and issue |
|
I also added sdist tests so that we can be sure that sdist can be build on specified os/python combinations: https://github.com/dudoslav/TileDB-Py/actions/runs/9303158461 |
| - name: Get sdist package name | ||
| id: get_sdist_name | ||
| run: | | ||
| echo "sdist_name=$(ls dist/ | head -n 1)" >> "$GITHUB_OUTPUT" |
There was a problem hiding this comment.
This is relying on the file ordering, seems like it could be brittle? Can we construct the sdist name from the version? Or at least it should be ls *.tar.gz to be more specific.
There was a problem hiding this comment.
There should be only one file in /dist since this is completely clean job. But adding *.tar.gz is a good idea.
| path: dist | ||
|
|
||
| - name: Install sdist artifact | ||
| run: pip install --verbose dist/${{ needs.build_sdist.outputs.sdist_name }} |
There was a problem hiding this comment.
Not blocking but we might want to change this later to run in parallel with the wheel builds, because this will have to build everything from source so it will take some time at the end that could be amortized.
There was a problem hiding this comment.
I am not sure if I understand correctly, but this does run in parallel right now. All jobs (around 40) run in parallel, half is building wheels and half is building sdist and testing it.
Rework build system into CMake + scikit-build-core combination. CMake is used to build all native bindings (pybind11/Cython) and to download TileDB core library if needed. Scikit-build-core is responsible for packaging this logic into python wheels and sdist.
Some CI/CD changes were required. We no longer need to download TileDB core library in CI/CD since CMake can do if for us. So all CI workflows are now reworked into two main ones:
ci.yml- build and test tiledb-py by installing it:pip install .and then running tests against this installbuild_wheels.yml- build wheels for each platform/python_version and then install wheel and test it.Both of these pipeline are very similar so it might make more sense to just use one, preferably
build_wheels.yml.As with all our other projects, this project also respects
TILEDB_PATHand uses it as if it wasTileDB_DIRfor interoperability with CMake'sfind_package.Note: Running tests locally is an issue right now, because tests need python bindings already compiled and installed in
tiledbfolder, which is a source folder.