Avoid coercing to numpy in as_shared_dtypes

[TomNicholas]

• Solves the problem in Only use CopyOnWriteArray wrapper on BackendArrays #8712 (comment)
• Tests added
• User visible changes (including notable bug fixes) are documented in whats-new.rst
• New functions/methods are listed in api.rst

[TomNicholas]

Previously this asarray call would coerce to numpy unnecessarily, when all we really wanted was an array type that we could examine the .dtype attribute of.

[TomNicholas]

Will this idea always work? What if it steps down through a lazy decoder class that changes the dtype...

[dcherian]

Those should be going through

xarray/xarray/coding/variables.py

Lines 52 to 64 in c9ba2be

	class _ElementwiseFunctionArray(indexing.ExplicitlyIndexedNDArrayMixin):
	"""Lazily computed array holding values of elemwise-function.
	Do not construct this object directly: call lazy_elemwise_func instead.
	Values are computed upon indexing or coercion to a NumPy array.
	"""
	def __init__(self, array, func: Callable, dtype: np.typing.DTypeLike):
	assert not is_chunked_array(array)
	self.array = indexing.as_indexable(array)
	self.func = func
	self._dtype = dtype

so you should be fine.

[TomNicholas]

I think I'm getting confused as to how this all works now... Don't I want to be computing as_shared_dtype using the dtype of the outermost wrapped class? Whereas this will step through all the way to the innermost duckarray, which may have a different dtype?

[dcherian]

As of now, as_shared_dtype is expected to return pure duck arrays for stack, concatenate, and where.

So that means we need to read from disk, which you do with to_duck_array and all these wrapper layers will be resolved.

It will get more complicated when we do lazy concatenation in Xarray, then we'd need to lazily infer dtypes and apply a lazy astype.

[TomNicholas]

Testing this is confusing me - I want to add an xr.Variable.concat to test_dataset.py::TestDataset:test_lazy_load_duckarray but the existing DuckArrayWrapper class does a funny trick where it defines __array_namespace__ but doesn't actually implement the xp namespace...

[dcherian]

Can we use the to_numpy in this file instead?

[dcherian]

This is fine but will force a read from disk. We could add a dtype property that forwards to the underlying array.dtype

EDIT: I don't think my comment is right, since we expect to return duck arrays here, it's ok to just read from disk and create that duck array.

It will get more complicated when we do lazy concatenation in Xarray, then we'd need to lazily infer dtypes and apply a lazy astype.

[TomNicholas]

When you say "read from disk" do you meaning calling the __array__ attribute of the innermost duckarray? Because that's what I'm trying to avoid.

EDIT: Or you mean resolving all these wrapper layers (either by calling __array__ or get_duck_array())?

[dcherian]

do you meaning calling the array attribute of the innermost duckarray?

I think our naming convention is that "duck array" is a "computational array" e.g.numpy, dask but NOT our explicitly-indexed array classes. The latter wrap duck arrays.

Read from disk should be happening by calling get_duck_array on the outermost ExplicitlyIndexed class, which should propagate down to BackendArray which reads bytes using either indexing or np.asarray (I think).

(related : zarr-developers/zarr-python#1603 (comment))

PS: We could chat on a call some time if you want. It's all quite confusing :) This is a good opportunity to add some comments/docs for devs

[TomNicholas]

Read from disk should be happening by calling get_duck_array on the outermost ExplicitlyIndexed class, which should propagate down to BackendArray which reads bytes using either indexing or np.asarray (I think).

Yes, but my KerchunkArray case is interesting because I don't want to use BackendArray (I have no use for CopyOnWrite because I'm never loading bytes, nor for Lazy indexing (I can't index into the KerchunkArray at all).

PS: We could chat on a call some time if you want. It's all quite confusing :) This is a good opportunity to add some comments/docs for devs

Yeah that could be helpful actually :) I'm learning a lot right now about a part of xarray I have never had a reason to look at before!

[dcherian]

but my KerchunkArray case is interesting because I don't want to use BackendArray

Well then don't use the backend infrastructure? :P

[TomNicholas]

Haha yes yes

No but seriously I did think about that and I do think that it does make sense to use the backend infrastructure here. I could make my full case, but after all we are still reading from files here, we just aren't reading the bytes inside the chunks.

[dcherian]

class does a funny trick where it defines array_namespace but doesn't actually implement the xp namespace...

It's faking to get past the checks in as_compatible_data.

[TomNicholas]

It's faking to get past the checks in as_compatible_data.

Yep, but doing it this way (instead of e.g. defining __array_function__) is violating the contract the array API represents, so I'm finding that get_array_namespace gets called on it (and causes errors after not finding the namespace). I haven't gotten to the bottom of this yet though.