Implement PythonCall extension to convert xarray objects

[JamesWrigley]

This is quite handy for interop with Python. There's a couple of things I might need a hand with:

• The tests create a 2D DataArray with only one dimension having a coordinate. When I run the test code in the REPL, lookup(y, :length) returns NoLookup, which is what I expect. But in the tests it seems to automatically create a lookup of 1:5 and I'm not sure where that's coming from because it's not being created by the pyconvert() methods.
• I couldn't figure out how to add a new page to the docs with the modifications in the PR. Not sure if I'm building them wrong or if there's something missing.

Also note that the tests create a shared conda environment named @dimensionaldata-tests, I went for a shared environment instead of a local one because this way it gets stored in the depot and will get cached by julia-actions/cache.

[JamesWrigley]

(I noticed that CI wasn't caching anything anyway so I took the liberty of adding it in d3c1730)

[felixcremer]

There is also https://github.com/meggart/PyYAXArrays.jl which might give some inspiration.

[rafaqz]

But in the tests it seems to automatically create a lookup of 1:5

Can you show where you mean? NoLookup is always just a Base.OneTo(length) underneath because a Lookup is an AbstractArray and it needs length, so may as well be the indices. That will sometimes display as a 1:5 vector

[rafaqz]

But this looks really nice to have overall.

Should we also handle/test more complicated lookups like DateTime? And is there information from xarray if a lookup is points or intervals, categorical etc?

[JamesWrigley]

Can you show where you mean? NoLookup is always just a Base.OneTo(length) underneath because a Lookup is an AbstractArray and it needs length, so may as well be the indices. That will sometimes display as a 1:5 vector

Ah I'm dumb... the test was checking if the lookup == NoLookup() instead of isa NoLookup.

Should we also handle/test more complicated lookups like DateTime?

Agreed that it'd be useful, but I do not currently have the courage to tackle that hornets nest 😅

And is there information from xarray if a lookup is points or intervals, categorical etc?

No I don't think that kind of information is supported yet according to the docs: https://docs.xarray.dev/en/stable/user-guide/data-structures.html#coordinates
But it may come in the future: pydata/xarray#8463

When reading that page I realized that these conversion methods won't work in two situations:

• Multiple coordinates/lookups assigned to a single dimension.
• Multi-dimensional coordinates/lookups that are assigned to multiple dimensions.

Does DimensionalData support those at all? From the docs I can't figure out how to construct a DimArray for either situation.

[rafaqz]

When reading that page I realized that these conversion methods won't work in two situations:
Multiple coordinates/lookups assigned to a single dimension.
Multi-dimensional coordinates/lookups that are assigned to multiple dimensions.
Does DimensionalData support those at all? From the docs I can't figure out how to construct a DimArray for either situation.

I'm not sure how common the first would be, or if I understand totally, what's the use case?

The second I have just never implemented but isn't too hard in theory, all the machinery is in place with Transformed lookups we "just" need to swap the functions for matrices. (But it's also very rare in the wild)

My guess is merging this without DateTime handling workingwill lead to a bunch of bug reports. But won't pythoncall just handle that for us? Did you try it?

[rafaqz]

Lets just merge and fix details later

[asinghvi17]

https://github.com/meggart/DateTimes64.jl was written for Python datetime -> Julia conversion.

[JamesWrigley]

My apologies for not getting back to this, I didn't mean to ghost you into merging it 😬 Things got a bit busy leading up to Christmas (happy new year BTW 🥳) and this slipped my mind. I'll try to tackle datetime conversion later in the week.

When reading that page I realized that these conversion methods won't work in two situations:
Multiple coordinates/lookups assigned to a single dimension.
Multi-dimensional coordinates/lookups that are assigned to multiple dimensions.

I'm not sure how common the first would be, or if I understand totally, what's the use case?

The second I have just never implemented but isn't too hard in theory, all the machinery is in place with Transformed lookups we "just" need to swap the functions for matrices. (But it's also very rare in the wild)

I actually use both semi-regularly. For the first case I often work with arrays of events that have a time and 'quality', i.e. a DataArray with a time coordinate and a boolean coordinate indicating whether the event is trustworthy or not.

For the second, I also often work with multi-dimensional arrays that have two different kinds of times: one is a coarse time and the other is orders of magnitude more precise but it's an offset relative to the coarse time. So to get the time of an event I have to combine the coarse time and offset. Concretely, the DataArray has dimensions (coarse_time, offset_time, event) and it'll have a 2D time coordinate that combines coarse_time and offset_time (it's important to keep coarse_time and offset_time as dimensions for selecting/aligning with other data).

In both cases having a separate array for the lookups would work, but it's really nice to keep them together. I'm not that familiar with the DimensionalData internals, but if you give me some pointers I'm happy to take a shot at implementing them?

[sethaxen]

Is there interest in also having a conversion from DimStack/DimArray to an xarray.Dataset, etc? This would allow users to work with Python packages that expect xarray inputs from Julia using DimStack, etc.

Edit: there's a very barebones implementation of this here: https://github.com/arviz-devs/ArviZPythonPlots.jl/blob/main/src/xarray.jl

[asinghvi17]

That would be great, there's another implementation for YAXArrays at https://www.github.com/meggart/PyYAXArrays.jl