xr.concat is very slow

[mortonjt]

This question was started by this stackoverflow thread.

Long story short, I'm noticing that xr.concat is crazy slow. I have roughly 45k Datasets that I want to concatenate on a new dimension, and it looks like this concat operation is single threaded and takes many hours to run (I haven't been able to run it to completion yet).

I'm not exactly sure what is causing the bottleneck here, or how to speed this up. I'd figure that xr.concat should be able to be parallelized, since this should boil down to one big reduce operation.

EDIT: weird, I'm noticing that if I only focus on 1k datasets, xr.concat is able to actually leverage dask, and complete within minutes.

[max-sixty]

One option is to write them out to files, and then use xr.open_mfdataset; check out the docs.

[mortonjt]

Thanks @max-sixty , I think that sort of strategy is the way to go.
My use case is slightly more complex, since I'm using arviz.InferenceData objects, each of which wraps multiple xr.Dataset objects. I'm able to lump the 45k az.InferenceData objects into 45 larger az.InferenceData objects, each with 1k objects, and this seems to be able to actually leverage dask parallelism and complete within 3 hours.

So I'm beginning to think this is may be more a problem with dask than xarray, where dask struggles to deal with parallelizing tens of thousands of tasks.