Support of numpy.uint64 as VipsBandFormat

[unidesigner]

Hi!
I'd like to convert a Numpy array of dtype uint64 to a Vips Image (or a TIFF image), but without success. I tried to use the bandformat 'ulong', but this does not seem to be supported

Error: no value ulong in gtype VipsBandFormat (94004864223568)
pyvips: enum 'VipsBandFormat' has no member 'ulong', should be one of: notset, uchar, char, ushort, short, uint, int, float, complex, double, dpcomplex

Do you have any suggestions how I could use uint64 images with Vips? Thanks.

[jcupitt]

Hello @unidesigner, sorry, it's 8/16/32-bit ints and uints only.

You can have 32/64/128-bit complex, if that's any help (I guess not).

[unidesigner]

Hi John, ok. I can split to 64 bit uint into two 32bit images, that should work. I could also try to encode the two 32-bit integers into a 64-bit complex image, but I'm not sure if this would support tiled, compressed storage (as I'm doing with uint 32-bit images at the moment) ? Thanks, Stephan

[jcupitt]

You could double the number of bands I guess. It might be neater than having two files.

You could use a double. They can store a 50-bit int, which might be enough.

For curiosity, why do you need uint64 pixels? I've only used uint32 occasionally and never run into range problems even there.

[unidesigner]

Oh, thanks for the hint regarding double, this might be sufficient indeed. I'll write you a PM regarding use case.

[cgohlke]

I'd like to convert a Numpy array of dtype uint64 to a Vips Image (or a TIFF image)

FWIW, tifffile should be able write a uint64 TIFF image.

[unidesigner]

Hi @cgohlke, thanks for the hint, I'll look into tifffile for my use case. Libvips has a very nice and efficient way to read lots of regions of interest fromtiled tiff (e.g. see #100) - can you comment on tifffile's ability to do something similar (multi-threaded)? Thanks.

[cgohlke]

Libvips has a very nice and efficient way to read lots of regions of interest fromtiled tiff (e.g. see #100) - can you comment on tifffile's ability to do something similar (multi-threaded)?

I did not mean to advocate tifffile as an alternative for pyvips, just as an easy way to write uint64 numpy arrays to TIFF. Tifffile does not support reading ROIs from TIFF images. Whole slide imaging is not a good use case for tifffile unless maybe you need/want the whole image as a numpy array in memory or a memory-mapped file for random access. For reading whole images/series as numpy arrays, tifffile uses multi-threading to decode tiles or read from multiple files. Since pyvips uses openslide, it supports many more slide formats than tifffile.

If pyvips turns out too slow for your specific case of multi-threaded reading small ROIs from large, tiled, jpeg2000 compressed TIFF files stored on NVMe from Python (IIUC), I would try implementing this specific case in Cython using OpenJPEG, OpenMP, and memory-mapping or operating system file IO (not C stdlib read). Still, it will be hard to beat vips at this.

[jcupitt]

openslide supports parallel decode of jp2k-compressed tiles, so there's probably not a huge win there.

libvips won't do parallel decode of jpeg-compressed tiled tiff since libtiff does not support it, so there's maybe an opportunity there. It's not easy to do, unfortunately.

[unidesigner]

Thanks for the advice. Do I understand correctly that writing a uint64 numpy array with tifffile wouldn't allow me to read the file again with pyvips? As you write, I might get away with memory-mapped multi-threaded small ROI reading.

I'd need lossless compression. I am not very familiar with the jpeg2000 format, but I read just now that the jpeg2000 standard also supports a lossless option. I'd have to investigate further what its trade-offs are compared to other (supported) compression codecs for my data.