"libarrow.so.800: cannot open shared object file: No such file or directory" when Arrow is not installed

[ttencate]

Expected behavior and actual behavior.

When GDAL is compiled with Arrow support (-DGDAL_USE_ARROW=ON), but libarrow.so is not available at runtime, any use of the library outputs ERROR 1: libarrow.so.800: cannot open shared object file: No such file or directory several times. It doesn't seem to affect functionality as long as one doesn't try to access Arrow datasets.

This is a problem for package maintainers because either they would either need to disable Arrow support entirely, or make Arrow a hard dependency even if GDAL is otherwise perfectly usable without this driver.

Instead, when built with Arrow support but Arrow is not found, the driver should silently become unavailable at runtime.

Steps to reproduce the problem.

• Build GDAL with Arrow support (-DGDAL_USE_ARROW=ON).
• Uninstall Arrow.
• Run any GDAL tool, e.g.:

$ gdalinfo --version
ERROR 1: libarrow.so.800: cannot open shared object file: No such file or directory
ERROR 1: libarrow.so.800: cannot open shared object file: No such file or directory
ERROR 1: libarrow.so.800: cannot open shared object file: No such file or directory
ERROR 1: libarrow.so.800: cannot open shared object file: No such file or directory
GDAL 3.5.1, released 2022/06/30

Operating system

Arch Linux, x86_64

GDAL version and provenance

gdal_3.5.1-2 from the official Arch packages.

Note that I originally reported this to the Arch package maintainers, but they believe (probably rightly) that Arrow is meant to be an optional dependency at runtime (if enabled at compile time), making this an "upstream issue".

[rouault]

I don't know how Arch packages GDAL, but if you get this error, it very likely means that they build the Arrow driver as a plugin, and this "ERROR 1: libarrow.so.800: cannot open shared object file: No such file or directory" is rightly emitted by the ogr_Arrow.so plugin since you uninstalled libarrow after building. As far as I can tell, this is the best behaviour GDAL can do. The Arrow driver should probably be packaged by Arch as a separate package, with a dependency to libarrow, so that when you uninstall (EDITED) libarrow, the Arrow driver is uninstalled.

[ttencate]

I did some further digging; indeed the Arrow driver is built as a plugin. Indeed, several other plugins have the same behaviour, but it turns out I had all these other optional dependencies already installed. If I uninstall, for example, netcdf and mariadb-libs, I get complaints about libnetcdf.so.19 and libmariadb.so.3 as well.

Handing this back to the Arch packagers. Thanks for the quick response!

[kgizdov]

@rouault I guess this is an interesting design choice made by GDAL team at some stage, but on the other hand we definitely expect most packages to still work if a plugin of theirs is missing or even crashes. The simplest solution for us at the moment is to include all dependencies as hard dependencies to make it work.

However, I'd be open to some concrete suggestions as to how to make the packaging work. I see Ubuntu (kinetic) repackages plugins as a separate package containing only the /usr/lib/x86_64-linux-gnu/gdalplugins/drivers.ini file. I presume without that file (which we ship under usr/lib/gdalplugins) GDAL would not (try to) load any of its plugins, thus not crash if a plugin is misconfigured, correct?

What files should be split out as a separate package? I did not find much documentation around the subject. Thanks in advance.

[rouault]

I presume without that file (which we ship under usr/lib/gdalplugins) GDAL would not (try to) load any of its plugins, thus not crash if a plugin is misconfigured, correct?

GDAL would manage to load plugins, even without drivers.ini. drivers.ini just indicate to GDAL in which order it must try to probe plugins when opening a file. If a plugin is in the plugin driver but not in the list, it will be probed before listed plugins. But from a packaging point of view, this file should be provided. Debian has in a separate package than the core library because of the x86_64-linux-gnu arch-specific part. If you install in /usr/lib/gdalplugins, drivers.ini should just be included in the package of the core library

What files should be split out as a separate package?

That's really a packager decision. Several factors might be taken into account: license of the dependencies of the driver, size of those dependencies, etc.
Documentation related to drivers built as plugins: https://gdal.org/development/building_from_source.html#build-drivers-as-plugins

[kgizdov]

@rouault the way GDAL handles plugin loading internally is not really a packager decision and the procedure is not detailed anywhere in the documentation. The question is - how do we have plugins compiled at compile time, but when some of them are not explicitly installed by a user, then the main application would still work and not be completely inoperable?

For example, would we have to ship the whole /usr/lib/gdalplugins directory as a separate package that would depends on all the plugin dependencies? Would that break GDAL? Would it suffice to GDAL_AUTOLOAD_PLUGINS:BOOL=OFF and GDAL then has some internal mechanism to allow users to manually load plugins at will?

So many unknown not listed in the docs. A packager can only make a decision only for how to implement what upstream demands but cannot read minds.

[rouault]

If a driver built as plugin at build time is not present in /usr/lib/gdalplugins, no issue: GDAL will not try to load it, and will not emit any error

would we have to ship the whole /usr/lib/gdalplugins directory as a separate package that would depends on all the plugin dependencies?

You could decide either to have a single "gdal-plugins" package with all the content of /usr/lib/gdalplugins, and with appropriate dependencies to the libraries required by those plugins
Or split it in "gdal-plugin-foo" with only /usr/lib/gdalplugins/gdal_foo.so in it and just the libs gdal_foo.so depend on as requirements, and a "gdal-plugin-bar" with only /usr/lib/gdalplugins/gdal_bar.so in it, etc.

Other documentation related to plugin loading: https://gdal.org/api/gdaldriver_cpp.html#_CPPv4N17GDALDriverManager15AutoLoadDriversEv

One example is the gdal-grass plugin. e.g https://packages.debian.org/sid/libgdal-grass (ok, that one is a bit particular as it is built separately of the OSGeo/gdal infrastructure)

conda-forge/gdal-feedstock#679 is an example of a work-in-progress to have a "llibgdal-arrow-parquet" package in conda-forge that bundles the ogr_Arrow.so and ogr_Parquet.so plugins.

[kgizdov]

Thanks for this!