Provide a bundled flag for gdal-sys
#517
Conversation
Cargo.toml
Outdated
|
|
||
|
|
||
| [patch.crates-io] | ||
| proj-sys = { git = "https://github.com/GiGainfosystems/proj", rev = "54716dd8955d4f0561ce9bf8a83610b605e3c007" } |
There was a problem hiding this comment.
This currently relays on a non-accepted patch to proj-sys: georust/proj#190
There was a problem hiding this comment.
I pushed 27c2a5e which changes this to use the master branch of proj-sys instead as the relevant patch is now merged there. There is still no release that contains this change yet.
There was a problem hiding this comment.
Seems like we still need georust/proj#192
There was a problem hiding this comment.
That's finally released with proj-sys 0.24 🎉
gdal-src/build.rs
Outdated
| .define("BUILD_GMOCK", "OFF") | ||
| .define("PROJ_INCLUDE_DIR", format!("{proj_root}/include")) | ||
| .define("PROJ_LIBRARY", format!("{proj_root}/lib/{proj_lib}")) | ||
| // enable the gpkg driver |
There was a problem hiding this comment.
I require the geopackage driver for my usecase that's why I added these configurations. In the end we already depend on libsqlite3 via proj, so that should hopefully be fine.
That written: Anyone that tries to actually use that driver to create a geopackage dataset will likely run into: OSGeo/gdal#9135 (which hopefully will be fixed soon upstream, so it might be worth to pull in the fix into the bundled version afterwards.)
|
Marked as draft as I need to perform a final set of tests on MacOS and Windows. |
| @@ -1,6 +1,7 @@ | |||
| # Changes | |||
|
|
|||
| ## Unreleased | |||
| - Add a `bundled` feature for `gdal-sys` that allows to build and statically link a minimal bundled version of gdal during `cargo build` | |||
There was a problem hiding this comment.
We should probably mention that it only supports SQLite, GPKG, GeoTIFF, some other less popular formats, and probably no compression. I'm afraid that users will see this and expect it to work for COG, JPEG2000, NetCDF and so on.
Also see https://www.mail-archive.com/[email protected]/msg40172.html.
There was a problem hiding this comment.
The following 19 drivers are supported by this configuration according to DriverManager::all():
Virtual Raster
Derived datasets using VRT pixel functions
GeoTIFF
Cloud optimized GeoTIFF generator
Erdas Imagine Images (.img)
In Memory Raster
Geographic Network generic file based model
Geographic Network generic DB based model
ESRI Shapefile
MapInfo File
VRT - Virtual Datasource
Memory
Keyhole Markup Language (KML)
GeoJSON
GeoJSON Sequence
ESRIJSON
TopoJSON
GeoPackage
SQLite / Spatialite
gdal-src/build.rs
Outdated
| "libproj.a" | ||
| }; | ||
|
|
||
| let res = cmake::Config::new("source") |
There was a problem hiding this comment.
For anyone wondering, cmake automatically sets CMAKE_BUILD_TYPE.
|
This looks all right, but as I mentioned in the related issue, I'm worried it's only going to be useful to you. GDAL is pretty large, and its users might have a lot of expectations, but a version that only supports GPKG is going to be very surprising. Even your use case might be better served by https://docs.rs/geozero. CC @rouault long-term, how do you feel about ExternalProject (SuperBuild)? |
weiznich
force-pushed
the
feature/bundled_build
branch
from
a30248e to
1803cfd
Compare
I agree with the point that users might have expectations that are not covered (yet) by the proposed bundling support, but I disagree with with the point that it only supports As for other drivers: It is certainly possible to enable other drivers as well. That mostly requires going through the list of supported drivers and classify them if they need external dependencies or not. The later ones can easily be enabled, while for the former ones more work is required. Depending on the required external dependency that might require adding another sys crate as dependency and enabling the bundling support there (like for example for
Trust me if I say that no geozero won't solve my use-case as it has to much assumptions around which geometry types exist and how they should be handled for my use-case. (It's specifically not about translating from one geometry format to another). |
|
Yes, I think it's all right. We should probably document our features anyway, and that's a good place where we can explain the limitations of the bundled GDAL. |
|
I just checked the documentation. GDAL already provides a list of dependencies for each driver: https://gdal.org/drivers/vector/index.html Anything that lists "Built-in by default" as dependency can probably just be enabled without problems. I also know that bundling support exists for the following other dependencies:
So yes, it's probably possible to just enable most of the expected drives, although I personally would prefer having all the native dependencies behind separate feature flags. |
weiznich
force-pushed
the
feature/bundled_build
branch
from
1803cfd to
1315e16
Compare
I miss some contextual elements to understand your question |
|
If you're not familiar with it, ExternalProject is a CMake component that can download libraries, apply patches, then link your project with them. So it can be useful for people who want a custom build but don't want to build every dependency manually. If GDAL adds that in the future, we can integrate with it without creating Rust libraries for that bundle the source code of the GDAL dependencies. If it doesn't, that's fine, people can still use GeoTIFF, GPKG, SHP and a a few other formats with the approach here. |
|
ok, I see. That's something I've considered, but I'd be hesitant to go into that business, as it equates to creating yet another packaging system, and for GDAL, with all its transitive dependencies, that can mean ~ 100 libraries for a full build... That said I do recognize that might be a recurring need for people in "unstandard" environments (Android, WASM, etc.) to have to rebuild the whole world. I'm not sure if that would belong to the OSGeo/GDAL repo itself, or if it might be a side repository where people from different teams can collaborate, and be independent of GDAL development itself. Might be worth raising the idea on gdal-dev. |
|
Yeah, that makes sense. You've already got a full plate with GDAL and the other libraries you're maintaining, you don't need to get into a whole new packaging system unless you really want to. It would only make a difference for this PR if you were already planning to add it. |
weiznich
force-pushed
the
feature/bundled_build
branch
from
1315e16 to
8db29ac
Compare
weiznich
force-pushed
the
feature/bundled_build
branch
from
8db29ac to
e1d7faf
Compare
| cmake = "0.1.50" | ||
|
|
||
| [features] | ||
| default = [] |
There was a problem hiding this comment.
It's up to discussion which drivers should be build by default as cargo feature flags. We cannot easily say build everything and allow users to opt out as cargo feature flags are additive.
There was a problem hiding this comment.
I guess NetCDF and Zarr could be interesting.
There was a problem hiding this comment.
The zarr driver so currently not not supported by the bundling. It would require figuring out how to bundle all the dependencies for that driver as well.
Netcdf is supported, as there is already a rust crate which provides bundling support and exposes the relevant information. Therefore it should be possible to enable it by default.
|
I've pushed a new version of the build script that adds feature flags for a lot of the built in drivers. If I enable all of them List with 162 driver namesNotably this is still missing support for anything that depends on It's up to discussion which of these drivers should be enabled by default. |
weiznich
force-pushed
the
feature/bundled_build
branch
from
e1d7faf to
54ef44f
Compare
|
I confirmed that this works on linux (x86_86), windows (msvc, x86_64) and macos (aarch64). |
…s before any depentend crate
weiznich
force-pushed
the
feature/bundled_build
branch
from
7db4695 to
4b494d2
Compare
|
@jdroenner Thanks for the response ❤️
I still would like to hear some details on this. |
|
IMO, the submodule is the most boring part of this. I don't like them, but the alternatives are worse (as you've clearly shown above). I'm more worried about the CMake config and proliferation of features, but we'll see how problematic they are. |
|
I'm attempting to use this PR in a project that also uses the geo package with the "use-proj" feature and getting version confilicts for proj_sys with the key part of the error being: Is there a way to specify which proj version to link to? |
|
@Isaac-Leonard cargo only allows one version of a |
|
I was able to fix it by patching dependencies to the proj repo and it works grate. |
|
@Isaac-Leonard It might be possible to do something like that, but at least for me that's clearly out of scope for this PR as it is additional functionality. I suggest that you check with the rust-gdal maintainers if they are comfortable with adding a new feature flag for building the binaries as well. After that you would need to change the |
|
Regarding the apps, I think we should instead offer bindings for them. It's not too complicated, just that there's a ton of API surface to bikeshed. And, as weiznich said, it's a bit unusual to build apps together with a library crate. That said, can we merge this now? |
|
yes lets merge it |
|
I'm naturally interested in seeing a release that contains this new feature. Is there anything that I could do to make this happen faster? |
|
Let's pick a reasonable cut-off date (say 2-4 weeks from now), in case we can get in any other PRs until then. Would that time frame work for you? |
|
That would be really nice. It's also fine for me if it takes a few weeks longer, but I would prefer not to have wait months for this, as cloning the submodule everytime in CI is a possible source of network failures. |
|
Or let's just publish it as -alpha.0 or something. |
This commit introduces a new
gdal-srccrate which bundles gdal and builds a minimal version from source viabuild.rsFixes #465
CHANGES.mdif knowledge of this change could be valuable to users.