Reduce dask tokenization time

[martindurant]

When using dask (e.g., chunks={} with a zarr dataset), each dask.array gets a token. Calculating this token currently hits a recursive path within dask and is relatively slow (~10ms), which adds up for many variables. This PR makes a simpler but still unique token.

An example profile of open_dataset before:

and after

[Illviljan]

This is a great catch @martindurant ! I've seen the tokenization slowness as well and this will be a great improvement!

Hmm, I was expecting this benchmark to get better:

xarray/asv_bench/benchmarks/dataset_io.py

Line 527 in 1e8f618

class IOReadCustomEngine:

Something to ponder for another day.

[martindurant]

Can you add a comment

Done.

I was expecting this benchmark to get better:

It would have an effect if n_chunk * n_variables is large (and only on the dask path)

[dcherian]

Well this benchmark didn't change.

xarray/asv_bench/benchmarks/dataset.py

Lines 23 to 32 in 8dddbca

	class DatasetChunk:
	def setup(self):
	requires_dask()
	self.ds = Dataset()
	array = np.ones(1000)
	for i in range(250):
	self.ds[f"var{i}"] = ("x", array)
	def time_chunk(self):
	self.ds.chunk(x=(1,) * 1000)

Can you propose an improvement?

[martindurant]

Can you propose an improvement?

That's for rechunk rather than open? I'm not sure it calls this code.

[dcherian]

_maybe_chunk is shared between the two code paths AFAICT.

[dcherian]

Hmmm I think I missed it earlier now I see

[52.71%] ··· dataset.DatasetChunk.time_chunk                            269±1ms # HEAD
[77.71%] ··· dataset.DatasetChunk.time_chunk                            402±1ms # main

and

[52.71%] ··· dataset.DatasetChunk.time_chunk                            375±2ms # HEAD
[77.71%] ··· dataset.DatasetChunk.time_chunk                            539±4ms # main

which is under the 50% reporting threshold we set but still great.