jp2 driver missing?

[bjoern234]

Hi,
I get this error message when I try to open a jp2 file:

ERROR: GDALError (CE_Failure, code 4):
	`L2A_T29UMV_20160926T115352_B02_10m.jp2' not recognized as a supported file format.

How can I fix this? I use Linux Ubuntu 16.04.
Thank you
Bjoern

[visr]

Yes this driver is missing from the GDALBuilder build currently. It would be some extra work to include it since it depends on external libraries being added as a dependency as well. However if there is enough demand we should probably try to do it.

Do you know which one of the drivers listed in https://www.gdal.org/formats_list.html it is (e.g. JPEG2000 or JP2OpenJPEG)?

If you cannot also easily get the files in for instance GeoTIFF format, it may be easiest to install your own GDAL with support for this format, and then in GDAL/deps/deps.jl update the paths to point to this GDAL instead.

[bjoern234]

The jp2 driver should be the OpenJPEG https://github.com/uclouvain/openjpeg/releases/tag/v2.3.0
I will try to change the GDAL version in the deps.jl... Thank you!

[cbdavis]

+1 for JP2 support

@bjoern234 did you manage to get things working?

My use case is that I'm trying to use GDAL.jl to read ESA Sentinel multi-spectral products, which are published as zip files. The version of GDAL packaged by GDALBuilder includes the Sentinel driver, which means that I can see the various files included in the zip archive, but without the JP2 driver, I can't actually read the individual images. While I could unzip the archive and convert the various jp2 files to geotiffs, this is quite bit slower than just using the GDAL drivers. While this isn't a big deal if you're just dealing with single observations, it adds up if you're like me and trying to process a few years of observations.

[cbdavis]

There's a --with-openjpeg parameter that can be used when configuring GDAL: https://wiki.orfeo-toolbox.org/index.php/JPEG2000_with_GDAL_OpenJpeg_plugin, and it seems that this could be used with the jp2 driver mentioned above.

[visr]

I think it'd be good to support it in the BinaryBuilder.

Right now I'm working on getting a working GDAL 3.0 build here: https://github.com/JuliaGeo/GDALBuilder/tree/gdal3

Most platforms seem to be working, but on FreeBSD I get an error configure: error: long long not found, and on Windows it for some reason does not find the PROJ 6.1 installation from BinaryBuilder. (https://travis-ci.org/JuliaGeo/GDALBuilder)

Since adding more drivers will only reduce the chance of a successful build, I prefer to first get a working GDAL 3.0 build out before adding more drivers.

[bjoern234]

@cbdavis as Martijn mentioned

it may be easiest to install your own GDAL with support for this format, and then in GDAL/deps/deps.jl update the paths to point to this GDAL instead."

In my case I installed GDAL via Conda (which includes the OpenJPEG driver) and changed const libgdal = joinpath(dirname(@__FILE__), "usr/lib/libgdal.20.dylib") to const libgdal = joinpath(dirname(@__FILE__), "/Users/bjoern/miniconda2/envs/py36/lib/libgdal.20.dylib")

[visr]

Ok great GDAL 3.0 seems to be working now.

If anyone wants to take this on, we first need an OpenJPEGBuilder that we can then add to the configure call and dependencies at https://github.com/JuliaGeo/GDALBuilder/blob/master/build_tarballs.jl.

Here are some instructions on building OpenJPEG:
https://github.com/uclouvain/openjpeg/blob/master/INSTALL.md
And here is an example of another GDAL dependency that uses CMake:
https://github.com/JuliaGeo/PROJBuilder

To create OpenJPEGBuilder it is the easiest to use the wizard, see https://github.com/JuliaPackaging/BinaryBuilder.jl and associated docs.

[cbdavis]

Is it better to build OpenJPEG from source instead of using a pre-compiled binary?

After my first post on this issue, I did some initial work that grabs the pre-compiled binaries (https://github.com/cbdavis/GDAL.jl/blob/gdal23/deps/build_OpenJPEG.v2.3.1.jl). There are still some bugs I need to fix, but it's able to download the files, although it doesn't extract them to the right directory.

I've been trying BinaryBuilder to build from source, although I'm still working through various issues (encrypted home drive issues on Linux, run into segmentation fault on Mac OSX). If using a pre-compiled binary is fine, I'll work on fixing up the existing code I have.

[visr]

Great that you're working on this!

I think that we want to setup and use a OpenJPEGBuilder with BinaryBuilder, instead of the pre-compiled libraries you linked to. (Of course with BinaryBuilder the user also downloads compiled code, it's just that we have to set up the Builder).

The pre-compiled OpenJPEG binaries come in 4 flavors, whereas we are currently providing 13 (these). I just downloaded the Windows zip, and I see that it is compiled with MSVC, whereas we use MinGW, this may introduce subtle issues. And if we need to modify the build a little, we can control directly how it is built.

It may be a bit more effort to set up the first time, but I think it will pay off to do it the way BinaryBuilder expects us to do it. If you get stuck with BinaryBuilder, perhaps you can put what you did so far online already? Then we can see if we can help to get unstuck.

[cbdavis]

I managed to get BinaryBuilder to work (https://github.com/cbdavis/OpenJPEGBuilder). I haven't tried to integrate this with GDAL yet, but at least when running build_tarballs.jl locally, I'm able to get 13 tarballs generated.

[bjoern234]

@cbdavis as Martijn mentioned

it may be easiest to install your own GDAL with support for this format, and then in GDAL/deps/deps.jl update the paths to point to this GDAL instead."

In my case I installed GDAL via Conda (which includes the OpenJPEG driver) and changed const libgdal = joinpath(dirname(@__FILE__), "usr/lib/libgdal.20.dylib") to const libgdal = joinpath(dirname(@__FILE__), "/Users/bjoern/miniconda2/envs/py36/lib/libgdal.20.dylib")

How can I link to a specific GDAL version since the deps.jl doesn't exists anymore in the new GDAL.jl version?

[visr]

Hmm I'm not sure what is the most convenient way now. You'd probably have to take the _jll dependencies out of the Project.toml, and give set the path in the code manually. But it's probably easier to do so on GDAL.jl 1.0 (before the switch, add [email protected].

Sorry I didn't get to integrate the OpenJPEG driver yet into the new build! It ended up taking quite some time before the base build was working. Now we should add more drivers here. I see
that since JuliaPackaging/Yggdrasil#904 OpenJPEG is already added to Yggdrasil, so doing that should be a matter of updating the dependencies and configure call in build_tarballs.jl.

[bjoern234]

Hi, thank you! Yes, it seems that it's more complicated than in previous versions. I will try to adapt the deps of the 1.0 version. I also tried to include the OpenJPEG_jll to generate new tarballs just by adding --with-openjpeg=$prefix to the ./configure block and OpenJpeg_jll to the dependencies. It seems to work but I'm not sure how to proceed, especially how to test if the tarballs created correctly.

[visr]

Oh, that's great that you already tried it! Testing them locally is a bit problematic, but there are quite some audits that check if it is correctly built, so if you don't get an error, that's a good sign. And you can search the build log for openjpeg to check if configure was indeed able to find and use the library.

Besides this, we should directly update the syntax in the build_tarballs.jl to the new form, compare sources and dependencies in the OpenJPEG and GDAL scripts, we need to wrap the strings in ArchiveSource, DirectorySource, and Dependency now.

And if you can wait a few days, perhaps it's worth waiting for the GDAL 3.1.0 release, which will likely be out around the weekend: https://lists.osgeo.org/pipermail/gdal-dev/2020-April/052015.html. Though it could be that we have to update the bundled patches in that case as well, so we could also do that later if you prefer not to change too many things at once.

[bjoern234]

I did not find any errors in the GDAL.log file, it seems to work. Next, I will try to adapt the syntax build_tarballs.jl as you suggested. For me it's fine to wait for the 3.1.0 release.

[visr]

FYI, I just got a ping that the GDAL 3.1.0 release is out: https://github.com/OSGeo/gdal/releases/tag/v3.1.0

[bjoern234]

I tried to build binaries with GDAL 3.1.0. For most Linux platforms the tarballs could built well but I got some errors building for aarch64 and powerpc64le which are seem related to libpng. For MacOSX and Windows the builds also failed. From my point of view it looks like a lot of work and it would be fine for me if we use GDAL 3.0.4 with openjpeg added for now.

[visr]

I see, thanks for checking. Could you perhaps put up a draft PR on Yggdrasil? If we don't figure it out quickly, we should indeed just add it to the current version. Or first put up a PR for 3.0.4 so that that can be released, and a draft PR of 3.1 after that so we can look at the issues. Thanks again!