GDAL utilities broken on Windows - apparent MbedTLS conflict

[niclasmattsson]

My packages that rely on GDAL utilities have stopped working. I suspected an unlinked DLL so I started experimenting with the utilities in Windows cmd. When I added all DLL folders from the artifacts to the path and ran gdalinfo I got this error:

Turns out the problem file was libmbedcrypto.dll. This DLL is in both MbedTLS_jll and the main bin folder of Julia 1.5.1 (on Windows at least). GDAL needs the one from MbedTLS_jll to take precedence, so either it needs to be copied to the current working dir or the path to it needs to be placed first in the library path.

julia> using GDAL

julia> GDAL.gdalinfo_path() do gdinfo
               run(`$gdinfo`)
           end
ERROR: failed process: Process(`'C:\Users\niclas\.julia\artifacts\1a3f285b91b22d9081d1ec6a0af713214f300fd6\bin\gdalinfo.exe'`, ProcessExited(3221225785)) [3221225785]
[snip]


julia> f = "libmbedcrypto.dll"; cp(joinpath(GDAL.GDAL_jll.LibCURL_jll.MbedTLS_jll.LIBPATH_list[1], f), joinpath(pwd(), f))
"C:\\Stuff\\Julia\\libmbedcrypto.dll"

julia> GDAL.gdalinfo_path() do gdinfo
               run(`$gdinfo`)
           end
Usage: gdalinfo [--help-general] [-json] [-mm] [-stats] [-hist] [-nogcp] [-nomd]
                [-norat] [-noct] [-nofl] [-checksum] [-proj4]
                [-listmdd] [-mdd domain|`all`] [-wkt_format WKT1|WKT2|...]*
                [-sd subdataset] [-oo NAME=VALUE]* datasetname

FAILURE: No datasource specified.
[snip]


julia> withenv("PATH"=>GDAL.GDAL_jll.LibCURL_jll.MbedTLS_jll.LIBPATH_list[1] * ";" * GDAL.GDAL_jll.LIBPATH) do
                run(`$(GDAL.GDAL_jll.gdalinfo_path_path)`)
           end
Usage: gdalinfo [--help-general] [-json] [-mm] [-stats] [-hist] [-nogcp] [-nomd]
                [-norat] [-noct] [-nofl] [-checksum] [-proj4]
                [-listmdd] [-mdd domain|`all`] [-wkt_format WKT1|WKT2|...]*
                [-sd subdataset] [-oo NAME=VALUE]* datasetname

FAILURE: No datasource specified.
[snip]

On a side note, please update the outdated README with an example on how to run GDAL utilites (#86). It's easy when you know how but figuring it out for the first time is really tricky, especially for users who don't yet understand how JLL packages and artifacts work. And having incorrect instructions up makes it even harder.

[visr]

Thanks for the detailed report! Since the issue seems to be mostly in differences between the MbedTLS build between julia/bin and MbedTLS_jll, I created an issue in Yggdrasil, JuliaPackaging/Yggdrasil#1679.

On a side note, please update the outdated README with an example on how to run GDAL utilites (#86).

Yeah I agree. When I created that issue I didn't know yet how to actually use them, and I have not used these GDAL command line tools since. Would be much appreciated if you could update this section! Hopefully this MbedTLS issue gets resolved soon.

[niclasmattsson]

Thanks for the update. I'll try to find some time to fix the README, but honestly what I should do is fix my own package which has a decent README but zero inline docs... :)

But for now I'll leave this here for other people who have problem with the GDAL utilities. This helper function contains the workaround that is needed until this issue gets fixed upstream:

function gdal_utility(f::Function, utility::String)
    artifactpath = GDAL.GDAL_jll.LibCURL_jll.MbedTLS_jll.LIBPATH_list[1]
    gdalcommand = getfield(GDAL.GDAL_jll, Symbol("$(utility)_path_path"))
    withenv("PATH" => "$artifactpath;$(GDAL.GDAL_jll.LIBPATH)") do
        f(gdalcommand)
    end
end

How to use it - first gdal_rasterize:

    shapefile = "combined-shapefile-with-oceans.shp"
    sql = "select FID+1 as FID from \"combined-shapefile-with-oceans\""
    outfile = "timezones.tif"
    options = "-a FID -a_nodata 0 -ot Int16 -tr 0.01 0.01 -te -180 -90 180 90 -co COMPRESS=LZW"
    gdal_utility("gdal_rasterize") do gdal_rasterize
        run(`$gdal_rasterize $(split(options, ' ')) -sql $sql $shapefile $outfile`)
    end

And here's ogr2ogr:

    shapefile = "combined-shapefile-with-oceans.shp"
    sql = "select FID+1 as FID,tzid from \"combined-shapefile-with-oceans\""
    outfile = "timezone_names.csv"
    gdal_utility("ogr2ogr") do ogr2ogr
        run(`$ogr2ogr -f CSV $outfile -sql $sql $shapefile`)
    end