Variable length data types

[LDeakin]

Resolves #21.

This is a substantial change that adds support for variable length data types to zarrs.
There were some breaking changes necessary to support this:

• Array store/retrieve "bytes" methods now take/return ArrayBytes which can represent fixed or variable length bytes, rather than just a slice-like
• Array store/retrieve "elements" variants use new Element[Owned] traits, with better validation
• Encoded bytes are aliased to RawBytes
• Codec traits have had some changes to accommodate the distinction between ArrayBytes and RawBytes

Data types

• String (utf-8)
• Binary

Codecs

vlen

{
  "name": "vlen",
  "configuration": {
    "data_codecs": [{"name": "bytes"},{"name": "blosc","configuration": {"cname": "zstd", "clevel":5,"shuffle": "bitshuffle", "typesize":1,"blocksize":0}}],
    "index_codecs": [{"name": "bytes","configuration": { "endian": "little" }},{"name": "blosc","configuration":{"cname": "zstd", "clevel":5,"shuffle": "shuffle", "typesize":4,"blocksize":0}}],
    "index_data_type": "uint32"
  }
}

Based on zarr-developers/zeps#47 (comment).

Structure:

• a little-endian uint64 representing the size in bytes of the encoded index,
• index data structured using the Apache arrow variable-size binary layout with the validity bitmap elided (https://arrow.apache.org/docs/format/Columnar.html#variable-size-binary-layout) and encoded with index_codecs,
• contiguous element bytes encoded with data_codecs.

The encoded index size is necessary to support index compression and partial decoding. If this were not available, the index could not used a bytes-to-bytes compression codec. A bytes-to-bytes compression codec could follow vlen, but then "data" is potentially running through a compression codec twice.

vlen_v2

{
  "name": "vlen_v2"
}

This matches Zarr V2 style interleaved encoding, which is implemented by numcodecs vlen-utf8, vlen-bytes, and vlen-array. These are all essentially the same codec, with data type-dependent behaviour. It makes sense to standardise a single codec for Zarr V3 to support Zarr V2 vlen-utf8/bytes/array encoded data without reencoding chunks.

Encoding Efficiency (32-bit index)

Sum of chunk sizes (in bytes) on "city" column of zarr-developers/zarr-python#2036 (comment).

https://github.com/LDeakin/zarrs/blob/variable_length_data_types/tests/cities.rs.

encoding	compression	size
vlen_v2		642196
vlen_v2	zstd 5	362626
vlen		642580
vlen	zstd 5	346950

[codecov]

Codecov Report

Attention: Patch coverage is 85.78135% with 424 lines in your changes missing coverage. Please review.

Project coverage is 81.33%. Comparing base (d54b89d) to head (71b9d78).

Files	Patch %	Lines
src/array/element.rs	65.41%	46 Missing ⚠️
...rray_to_bytes/sharding/sharding_partial_decoder.rs	88.28%	39 Missing ⚠️
src/array/array_sync_sharded_readable_ext.rs	63.95%	31 Missing ⚠️
...en_interleaved/vlen_interleaved_partial_decoder.rs	53.03%	31 Missing ⚠️
src/array/codec/array_to_bytes/vlen.rs	71.13%	28 Missing ⚠️
src/array/array_bytes.rs	93.63%	25 Missing ⚠️
...o_bytes/vlen_interleaved/vlen_interleaved_codec.rs	78.72%	20 Missing ⚠️
src/array/array_representation.rs	56.75%	16 Missing ⚠️
src/array/codec/array_to_bytes/vlen_interleaved.rs	68.00%	16 Missing ⚠️
src/array/array_async_readable_writable.rs	68.08%	15 Missing ⚠️
... and 27 more

Additional details and impacted files

@@            Coverage Diff             @@
##             main      #40      +/-   ##
==========================================
+ Coverage   79.56%   81.33%   +1.76%     
==========================================
  Files         142      152      +10     
  Lines       19544    20837    +1293     
==========================================
+ Hits        15550    16947    +1397     
+ Misses       3994     3890     -104     

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.