open_virtual_mfdataset

[TomNicholas]

Should we add a top-level function open_virtual_mfdataset?

Though I would like to be more confident about the best way to parallelize reference generation first

#123

#7

[TomNicholas]

Looking at the implementation of xr.open_mfdataset, I'm becoming optimistic that we can parallelize reference generation for concatenation along any number of dimensions just using dask.delayed or lithops.map. Basically we just use the same machinery xarray does for organising the open_dataset calls into a 1d list, optionally map over that in parallel, return the (small) virtual datasets back to the client over the network, then the multidimensional concatenation occurs on the client.