write numpy array to zarr with CompatCompressor

[kanglcn]

Hi, I am translating some of my numpy code to cupy. I used to store intermediate data to zarr with the default compressor and it works good. Since I want to use cupy now, using kvikio to accelerate the io is a good idea. But I can't translate all of them in short time. In some case, I need to do numpy processing and store the data and next read it directly to GPU memory with kvikio.zarr. What is the proper way to save numpy array to zarr with GPU compatible compressor?

I have tried to use

import zarr
import numpy as np
import kvikio
import kvikio.zarr
a = np.arange(20)
z = zarr.open('./tmp.zarr','w',shape=a.shape,dtype=a.dtype,chunks=a.shape,compressor=kvikio.zarr.CompatCompressor.lz4())
z[:] = a

But I got

BadCompressorError: bad compressor; expected Codec object, found <kvikio.zarr.CompatCompressor object at 0x1554c019f5b0>

[madsbk]

You should be able to use kvikio.zarr.CompatCompressor.lz4().cpu:

import zarr
import numpy as np
import kvikio
import kvikio.zarr

a = np.arange(20)
z = zarr.open(
    "./tmp.zarr",
    "w",
    shape=a.shape,
    dtype=a.dtype,
    chunks=a.shape,
    compressor=kvikio.zarr.CompatCompressor.lz4().cpu,
)
z[:] = a

I have added an example of this in #312

[kanglcn]

Thanks!

[kanglcn]

Another related issue, is it possible to write a zarr with compatcompressor in an environment without cuda? In some case I want to use CPU node (kvikio is not available) for some processing and write the intermediate result and then read them in GPU node and process with cupy.

[madsbk]

You can also use numcodecs.LZ4() directly.

[kanglcn]

Thanks.