Add array storage helpers

[d-v-b]

This PR adds nchunks, nbytes, and nchunks_initialized functionality from 2.x.

closes #2027
depends on #2064

details

Adds the following to array.py:

• (AsyncArray / Array).nchunks : deprecated, the total number of chunks in the array. exists for 2.xx compatibility.
• (AsyncArray / Array).cdata_shape : deprecated, the shape of the chunk grid. exists for 2.xx compatibility.
• (AsyncArray / Array).nbytes : the total number of bytes that the array can store
• (AsyncArray / Array)._iter_chunk_coords : an iterator over tuples of ints which represent positions in the chunk grid
• (AsyncArray / Array)._iter_chunk_regions : an iterator over slices which represent the contiguous array region spanned by each chunk
• (AsyncArray / Array)._iter_chunk_keys : an iterator over strings which represent the paths in storage for all the chunks
• chunks_initialized(array): a function that takes an array and returns a tuple of the chunk keys for that array that exist in storage. this also has tests.
• nchunks_initialized(array): deprecated, a function that calls len(chunks_initialized(array)). this exists for 2.xx compatibility.

All of the above _iter_chunk_* methods should be considered private and provisional. I added them because their functionality is valuable, but eventually I think we will have a better array API that renders these methods obsolete. If we think these are cluttering the array API, I'd be happy splitting them off into stand-alone functions.

• adds a function iter_grid to indexing.py, this just provides lexicographic iteration over the elements of a bounded N-dimensional, positive grid (e.g., a grid of chunks).

TODO:

• Add unit tests and/or doctests in docstrings
• Add docstrings and API docs for any new/modified user-facing classes and functions
• New/modified features documented in docs/tutorial.rst
• Changes documented in docs/release.rst
• GitHub Actions have all passed
• Test coverage is 100% (Codecov passes)

[d-v-b]

@tomwhite let me know if this looks workable for you

[tomwhite]

Thanks @d-v-b this looks great!

I wondered why you deprecated nchunks (and nchunks_initialized) though? The number of chunks in an array is something that should always be well-defined. Also, deprecating something usually means there's a better alternative, but I don't see one here.

[d-v-b]

I wondered why you deprecated nchunks (and nchunks_initialized) though? The number of chunks in an array is something that should always be well-defined. Also, deprecating something usually means there's a better alternative, but I don't see one here.

my thinking for this is twofold:

• with the new chunks_initialized function that gives the names of the initialized chunks, one can easily do len(chunks_initialized(...)), i.e. we don't need a separate function to express the composition of chunks_initialized and len. similarly, nchunks is merely len(array._iter_chunk_keys). If this logic is unsound, or these deprecation warnings are a problem, then we can remove them, but see the second point:
• we haven't yet figured out how we are going to express sharded arrays in the top-level array API, and I think those decisions might require rethinking how we express chunking more broadly. see conversations happening in this discussion. Until we solve that problem, I don't feel comfortable committing to any "here's what your chunks are like" APIs, especially if they are APIs that developed pre-sharding. hence adding private methods in this PR, and the deprecation warnings.

does this check out? I'm sorry if the warnings are inconvenient, but I really would like to find a proper expression of v3 semantics on the Array class and I worry that a blanket policy of forward-propagating v2-isms could be a hindrance to that effort.

[d-v-b]

The number of chunks in an array is something that should always be well-defined.

to expand on this: v3 introduces two kinds of chunks, read-chunks and write chunks. the number of read chunks may not equal the number of write chunks. so where we had 1 nchunks quantity in v2, v3 has two possible answers to nchunks. that's why it is not straightforward to commit to this aspect of the array API.

[jhamman]

@d-v-b - First of all, thanks for working on this! I must say I've soured a bit on deprecating some of these interfaces for the 3.0 release unless we have something to replace them. There's nothing to keep us from deprecating these in 3.1 (for example) if we come up with a new interface. What do you think about removing the warnings for now and coming back to this with a new interface down the road?

---

The number of chunks in an array is something that should always be well-defined.

to expand on this: v3 introduces two kinds of chunks, read-chunks and write chunks. the number of read chunks may not equal the number of write chunks. so where we had 1 nchunks quantity in v2, v3 has two possible answers to nchunks. that's why it is not straightforward to commit to this aspect of the array API.

I understand that we're trying to incorporate sharding here. At the risk of opening up a big can of worms, I think a we may be taking this too far. To me its much easier to think of chunks as the minimal block of data. Beyond that, sharding may allow you to store many chunks in a single object.

[jhamman]

I'm curious which of these helpers could migrate to the purview of the chunk grid.

[d-v-b]

What do you think about removing the warnings for now and coming back to this with a new interface down the road?

That works for me!

I understand that we're trying to incorporate sharding here. At the risk of opening up a big can of worms, I think a we may be taking this too far. To me its much easier to think of chunks as the minimal block of data. Beyond that, sharding may allow you to store many chunks in a single object.

Noted, I will pull back from the brink and make things more v2-ish again :)

[d-v-b]

@jhamman take a look when you have time, I think I addressed your concerns.

[d-v-b]

I should point out that this PR also contains some changes unrelated the the array API, but I think they are useful improvements:

• adds a _get_many method to all stores (the default implementation calls get in a loop). Stores that support any kind of batching / streaming can override this method as needed.
• defines a ByteRangeRequest type and ensures that all functions / methods that work with byte ranges are typed with ByteRangeRequest. Previously, we had inconsistent typing for that parameter across store methods and implementations.