Legate HDF5 using kerchunk

[madsbk]

Support reading of HDF5 files using kerchunk.

Limitations

• Only read is supported.
• Can only read whole arrays -- no partial access.
• No compression -- only HDF5 files written without compression is supported.
• Groups not supported -- only access to individual Zarr arrays are implemented.

Example

import h5py
import numpy as np

import legate_kvikio.kerchunk
import legate_kvikio.zarr


def hdf5_io(filename):
    a = np.arange(10000).reshape((100, 100))

    # Write array using h5py
    with h5py.File(filename, "w") as f:
        f.create_dataset("mydataset", chunks=(10, 10), data=a)

    # Read hdf5 file using legate+kerchunk
    b = legate_kvikio.kerchunk.hdf5_read(filename, dataset_name="mydataset")

    # They should be equal
    assert (a == b).all()


if __name__ == "__main__":
    hdf5_io("/tmp/legate-kvikio-io.hdf5")

[wence-]

Some minor suggestions and a few queries.

[wence-]

Maybe translate to zero-origin up front? (Does this do the right thing if point_dp is not in the box described by lo and hi?)

legate::Point<DIM> point = point_dp - lo_dp;
...
for (...) {
   idx = idx * extents[dim] + points[dim];

[madsbk]

Done.
point_dp must be within lo and hi. Since they come from context.get_task_index() and context.get_launch_domain() directly , I don't we need a check here.

[madsbk]

Thanks @wence- , I think I have addressed all of your comments.

[wence-]

Thanks @madsbk , one required correctness-checking change and I've now had a chance to go through the C++ task description and have one more query there.

[madsbk]

Thanks @wence-!

[wence-]

I think there was one "fixed" that you didn't push, but looks good, thanks

[madsbk]

Thanks @wence- !

[madsbk]

/merge