Support sharding for remote datasets

[fm3]

Both Neuroglancer Precomputed and Zarr (v3) can have sharded data formats. When served from a file server, range requests should be used to fetch only individual chunks out of the shards.

This needs to

• understand the sharding metadata of the datasets to figure out byte offsets and lengths of the chunks to request
• adapt the NIO file systems to support such partial reads
• adapt the data reading code to use those partial reads (both in local and remote case)

Possibly related:
https://stackoverflow.com/questions/35745403/java-how-to-read-part-of-file-from-specified-position-of-bytes
My assumption is that in the NIO implementation this will use SeekableByteChannel, which is implemented by the different NIO File System implementations. The HttpsFileSystem currently reads the whole file before returning a SeekableByteChannel, needs to be adapted to use http range requests. I don’t know how the S3 and GCS File System adapters behave.

[frcroth]

S3FileSystem seems to download the entire file and then return a Seekable channel on that file, so that wouldn't work for sharding.

[fm3]

Yeah, I hope we can change that, maybe the S3 client used by the s3fs has a specialized kind of GetObjectRequest we could use

[fm3]

I think a good first step would be to adapt the https file system to also see how well the NIO api works with the range requests. Feel free to exchange the http library used there if needed (I think the one I plugged in there has been discontinued in the meantime)