Dataset.__repr__ computes dask variables

[crusaderky]

DataArray.__repr__ and Variable.__repr__ print a placeholder if the data uses the dask backend.
Not so Dataset.__repr__, which tries computing the data before printing a tiny preview of it.
This issue is extremely annoying when working in Jupyter, and particularly acute if the chunks are very big or are at the end of a very long chain of computation.

For data variables, the expected behaviour is to print a placeholder just like DataArray does.
For coords, we could either

• print a placeholders (same treatment as data variables)
• automatically invoke load() when the coord is added to the dataset - see open_mfdataset reads coords from disk multiple times #1521 for discussion.

[shoyer]

I think we should probably just print the placeholders for dask arrays in all cases. The current behavior should require calling a .preview() method of some sort.

[crusaderky]

working on this now

[crusaderky]

This is in Jupyter:

def kernel():
    print("Kernel invoked!")
    return numpy.array([100, 200])

data = dask.array.Array(name='foo', dask={('foo', 0): (kernel, )}, chunks=((2,),), dtype=float)

ds = xarray.Dataset(
    data_vars={
        'foo': xarray.DataArray(data, dims=['x'], coords={'x': [10, 20]}),
        'bar': xarray.DataArray([1, 2], dims=['x'], coords={'x': [10, 20]}),
    })
ds.coords['y'] = ('y', [3, 4])
ds.coords['x2'] = ('x', data)
ds

Output:

Kernel invoked!
Kernel invoked!
Kernel invoked!
Kernel invoked!
Kernel invoked!
Kernel invoked!
Kernel invoked!
Kernel invoked!
Kernel invoked!
Kernel invoked!
Kernel invoked!
Kernel invoked!
Kernel invoked!
Kernel invoked!
Kernel invoked!
Kernel invoked!
Kernel invoked!
Kernel invoked!
Kernel invoked!
Kernel invoked!
Kernel invoked!
Kernel invoked!
Kernel invoked!
Kernel invoked!
Kernel invoked!
Kernel invoked!
Kernel invoked!
Kernel invoked!
Kernel invoked!
Kernel invoked!
Kernel invoked!
Kernel invoked!
Out[2]:
<xarray.Dataset>
Dimensions:  (x: 2, y: 2)
Coordinates:
  * x        (x) int64 10 20
  * y        (y) int64 3 4
    x2       (x) float64 100 200
Data variables:
    foo      (x) float64 100 200
    bar      (x) int64 1 2

YIKES!
Here you have two separate issues: one is with repr() as expected. The other is the fact that Jupyter invokes getattr() on a plethora of _ipython_* attributes that xarray doesn't define, and there was an issue where all non-index coords were blindly being converted to indexVariable (which loads the data) every. single. time.

[crusaderky]

There is a separate problem where index coords are computed twice. Didn't fix it yet and I am afraid of a domino effect. The problem is in merge.py:merge_coords():

    _assert_compat_valid(compat)
    coerced = coerce_pandas_values(objs)
    aligned = deep_align(coerced, join=join, copy=False, indexes=indexes)
    expanded = expand_variable_dicts(aligned)
    priority_vars = _get_priority_vars(aligned, priority_arg, compat=compat)
    variables = merge_variables(expanded, priority_vars, compat=compat)
    assert_unique_multiindex_level_names(variables)

Here, both expand_variable_dicts() and _get_priority_vars() compute the dask array.

[shoyer]

There is a separate problem where index coords are computed twice.

Index coordinates (i.e., IndexVariable) are supposed to be calculated once, when they are added to an xarray object.

If they are getting computed via as_variable in expand_variable_dicts (which is called from _get_priority_vars()) then something has already gone wrong.

[crusaderky]

Given this:

def kernel():
    print("Kernel invoked!")
    return numpy.array([100, 200])

data = dask.array.Array(name='foo', dask={('foo', 0): (kernel, )}, chunks=((2,),), dtype=float)

This correctly computes the coord once:

ds = xarray.Dataset(coords={'z': ('z', data)})

While this computes it twice:

ds = xarray.Dataset()
ds.coords['z'] = ('z', data)

[shoyer]

Hmm. My guess (untested) is that ds.coords['z'] = ('z', data) may be creating a Variable object instead of an IndexVariable by mistake. I opened another issue for this one: #1533

[crusaderky]

Travis is failing in a few environments, but I just tested that I get the exact same errors in the master branch