Libtiff issue while building --with-jepgli

[euzada]

Hi @jcupitt and @kleisauke,

I have a python script to stich images and save the final image into tiff using:

im.tiffsave(output_filename + extension, compression="jpeg", bigtiff=True, tile=True,
                    tile_width=256, tile_height=256, 
                    pyramid=True)

I build libvips V 8.15.2 with jepg-turbo and jpegli. For jpeg-turbo, the stichwork and the final image is generated with no issue. But if I use jpegli instead, I got an error.

Assertion failed: sp->cinfo.comm.is_decompressor, file /var/tmp/tmp-tiff-x86_64-w64-mingw32.shared.posix.web/tiff-4.6.0/libtiff/tif_jpeg.c, line 1115

I tested V8.16, and the same issue occurred.
or

Assertion failed: sp->cinfo.comm.is_decompressor, file /var/tmp/tmp-tiff-x86_64-w64-mingw32.shared.web/tiff-4.7.0/libtiff/tif_jpeg.c, line 1089

Any idea what can cause the issue? looking into libtiff, I beleive the issue could be from the following lines:
libtiff.

[euzada]

Hi @kleisauke,
I investigated this error and realized it is not related to tiffsave. Instead, the issue stems from my input images, which are TIFF files with JPEG compression. The error occurs while reading these files, not saving them. It seems that libtiff is unable to decompress the images. This only happened for the build --with-jpegli. Any idea?

[kleisauke]

Performing a round-trip conversion from JPEG to TIFF using a binary built with --with-jpegli works without any issues here:

$ vips copy zebra.jpg x.tiff[compression=jpeg]
$ vips copy x.tiff x.jpg

I did not encounter such issues in the libvips test suite either.

Did you completely remove the build/mxe/usr/<TARGET> directory before building with --with-jpegli or --with-jpeg-turbo? This ensures all dependencies are correctly built and linked against the appropriate libjpeg62-compatible DLL.

Also, what type of binary did you build? IIRC, I only tested the jpegli integration with the shared builds.

[euzada]

Thank you for your reply. I build mine with default like that which is web, shared and x86_64:
./build.sh --with-jpegli

I removed /build-win64-mxe completely before clone it again and rebuilding it. I get "vips-dev-w64-web-8.16.0-jpegli.zip".

Do you mind sharing with me your binary?

Regards,

[kleisauke]

Hmm, can you reproduce this error with the newly created binaries? I'm using the exact same build command, so I don't think there's any difference between these binaries.

[euzada]

I rebuild it right now and get "vips-dev-w64-web-030cbf9-disp-jpegli.zip". I added --with-disp and --nightly for the latest update. When I test it, I get again:

Assertion failed: sp->cinfo.comm.is_decompressor, file /var/tmp/tmp-tiff-x86_64-w64-mingw32.shared.disp.web/tiff-4.7.0/libtiff/tif_jpeg.c, line 1089

checking the libtiff website, here is the line where the error occured:

[tiff_jpeg.c line1089](https://gitlab.com/libtiff/libtiff/-/blob/master/libtiff/tif_jpeg.c?ref_type=heads#L1089)

This error again is about my input images that the libtiff had hard time to decompress the files. If I build it without jepgli, with turbo or mozjpeg, it works no issues. The libtiff can read my input files. I tested to read one file as input created as tiff with jpeg compression for a already stich image using turbo-jpeg, and it read it and save it with jpegli with no issue. except I got this warning:
VIPS-WARNING **: 09:30:43.968: Auto-corrected former TIFF subsampling values [2,2] to match subsampling values inside JPEG compressed data [1,1]

But if I tried to use the input images that was taken by microscope, it failed with assertion failed mentioned above.

Do you think my input images has issues but only when libtiff build with jpegli?

[kleisauke]

Are you able to provide a sample image that allows someone else to reproduce? If not, could you run tiffinfo on the image and send me the output of that?

Do you think my input images has issues but only when libtiff build with jpegli?

It sounds like a potential jpegli issue. I'll contact the JPEG XL Discord to check if there are any known compatibility problems with libtiff.

[euzada]

I have 2 set of images:
first set:

=== TIFF directory 0 ===
TIFF Directory at offset 0x3c5e12 (3956242)
  Image Width: 5488 Image Length: 3672
  Bits/Sample: 8
  Compression Scheme: JPEG
  Photometric Interpretation: RGB color
  FillOrder: msb-to-lsb
  Orientation: row 0 top, col 0 lhs
  Samples/Pixel: 3
  Rows/Strip: 3672
  Planar Configuration: single image plane
  Software: National Instruments IMAQ
  JPEG Tables: (289 bytes)

second set:

=== TIFF directory 0 ===
TIFF Directory at offset 0xb1fa (45562)
  Subfile Type: (0 = 0x0)
  Image Width: 5488 Image Length: 3672
  Resolution: 96, 96 pixels/inch
  Bits/Sample: 8
  Compression Scheme: LZW
  Photometric Interpretation: RGB color
  Samples/Pixel: 3
  Rows/Strip: 3672
  Planar Configuration: single image plane
  Predictor: horizontal differencing 2 (0x2)

it is very weird. If I do

vips copy first_image.tif output_first.tif[compression=jpeg,tile,tile-width=256,tile-height=256,pyramid,background=0]

or

vips copy second_image.tif second_first.tif[compression=jpeg,tile,tile-width=256,tile-height=256,pyramid,background=0]

they both works and load and save the output images correctly. Than why in python code I am having this error? I am very confused.

BTW, this test is on VIPS-8.17.0 that I build today.

Thank you.

[kleisauke]

What's the output of the following command?:

$ python -c "import pyvips; print('.'.join(str(pyvips.version(i)) for i in range(3)))"

[euzada]

What's the output of the following command?:

$ python -c "import pyvips; print('.'.join(str(pyvips.version(i)) for i in range(3)))"

8.17.0

[euzada]

Hi, I run in PowerShell a script to load each file and covert it using vips copy

$inputFile = $_.FullName
$outputFile = [System.IO.Path]::ChangeExtension($_.FullName, ".tiff")
vips copy '$inputFile' '$outputFile[compression=jpeg,tile,tile-width=256,tile-height=256,pyramid,background=0]'

than I run my python script on the new saved tiff files. This time I got a different error more detailed when I tried to use mosaic feature between 2 images. It indicates Wrong JPEG library version. I hope this can help. I tested also with V8.15.5 --with-jepgli and get the same error.

pyvips.error.Error: unable to call mosaic
  VipsJpeg: Wrong JPEG library version: library is 90, caller expects 62
tiff2vips: decompress error tile 0 x 0
VipsJpeg: Wrong JPEG library version: library is 90, caller expects 62
tiff2vips: decompress error tile 0 x 256
VipsJpeg: Wrong JPEG library version: library is 90, caller expects 62
tiff2vips: decompress error tile 0 x 512
VipsJpeg: Wrong JPEG library version: library is 90, caller expects 62
tiff2vips: decompress error tile 0 x 768

[kleisauke]

Ah, libjpeg(-turbo) and MozJPEG include safeguards to prevent version mismatches between the library and the caller. This ensures that the version used at runtime matches the version with which the library was originally built.

IIRC, jpegli doesn't implement such checks, so this issue seems like a DLL conflict at runtime. You might want to check your system for duplicates of libjpeg.dll or libjpeg-62.dll. You could try to use https://github.com/lucasg/Dependencies for this.

[kleisauke]

Hold on a moment. Do you see libjpeg.dll or libjpeg-62.dll in the bin/ directory? The latter should typically be the one generated, but it seems like jpegli is building libjpeg.dll instead. I'll work on a patch to align this, maybe that will fix it.

[kleisauke]

Commit 0f648d2 ensures that jpegli produces an output binary named libjpeg-62.dll and addresses issues related to static builds. To create a static binary, you can use the following command:

$ ./build.sh web x86_64 static --with-jpegli

[euzada]

hi @kleisauke, you are right, I do have libjpeg.dll instead of libjpeg-62.dll.

I tried to rebuild it again but I got error building libheif:

Failed to build package libheif for target x86_64-w64-mingw32.static.web!
------------------------------------------------------------
make[3]: *** [CMakeFiles/Makefile2:197: heifio/CMakeFiles/heifio.dir/all] Error 2
make[3]: Leaving directory '/var/tmp/tmp-libheif-x86_64-w64-mingw32.static.web/libheif-1.19.3.build_'
make[2]: *** [Makefile:156: all] Error 2
make[2]: Leaving directory '/var/tmp/tmp-libheif-x86_64-w64-mingw32.static.web/libheif-1.19.3.build_'
make[1]: *** [Makefile:913: build-only-libheif_x86_64-w64-mingw32.static.web] Error 2
make[1]: Leaving directory '/data/mxe'
real    0m20.856s
user    1m39.705s
sys     0m4.332s
------------------------------------------------------------
[log]      /data/mxe/log/libheif_x86_64-w64-mingw32.static.web

I saw you updated this lib too. Do you think it is related to the update?

Regards,

[kleisauke]

That libheif build error ought to be fixed via commit 0f648d2. What's the output of the following command in the build-win64-mxe directory?:

$ git rev-parse HEAD

[euzada]

That libheif build error ought to be fixed via commit 0f648d2. What's the output of the following command in the build-win64-mxe directory?:

$ git rev-parse HEAD

bc0167c

[euzada]

I tried to build it as shared, it works the build. No issue with libheif build. Only while I build it with static. But again it creates libjpeg.dll

[euzada]

I believe I get the latest update now. I am building it again:
Now it gives me the right commit.

$ git rev-parse HEAD
0f648d2063288cdabb037ba031d276b0f1ad1d4a 

[kleisauke]

To save you some build time, I'm (temporarily) providing prebuilt static binaries with jpegli here:
https://libvips-packaging.s3.amazonaws.com/vips-dev-w64-web-8.16.0-static-jpegli.zip

(these binaries were built from commit 0f648d2 using ./build.sh web x86_64 static --with-jpegli)

[euzada]

To save you some build time, I'm (temporarily) providing prebuilt static binaries with jpegli here: https://libvips-packaging.s3.amazonaws.com/vips-dev-w64-web-8.16.0-static-jpegli.zip

(these binaries were built from commit 0f648d2 using ./build.sh web x86_64 static --with-jpegli)

Thank you @kleisauke, I confirm that this patch works. No more Assertion failed error.

But as any 8.16 version, it takes 3 times longer than 8.15 version (similar time to jpeg-turbo in 8.16) but this is another issue that has PR in libvips.

Please feel free to close it.

[kleisauke]

Great! I'll close this issue for now, please feel free to re-open if there's still a problem.

Note that I couldn't reproduce the potential performance regression. It would be helpful if you could provide a minimal, (standalone) code sample that allows someone else to reproduce.