Having trouble with time dim of CMIP5 dataset

[jsbj]

Hi guys,

I was trying to use xray to look at CMIP5 data (e.g. https://dl.dropboxusercontent.com/u/1392864/rlut_Amon_BCC-bcc-csm1-1_rcp85_r1i1p1.nc), and the time dimension seems to fail. If I call

>>> dset["time.year"]
  ...
  File "/usr/local/lib/python2.7/site-packages/xray/core/dataset.py", line 59, in _get_virtual_variable
    data = getattr(date, var_name)
AttributeError: 'Index' object has no attribute 'year'

And thereafter I get

>>> dset
  ...
  File "/usr/local/lib/python2.7/site-packages/xray/core/indexing.py", line 371, in __array__
    return self.array.values.astype(dtype)
TypeError: Cannot cast datetime.date object from metadata [D] to [ns] according to the rule 'same_kind'

Is this all because the run goes to 2300?

Thanks!
Jonah

[darothen]

Hi @jsbj,

The fancy indexing notation you're trying to use only works when xray successfully decodes the time dimension. As discussed in the documentation here, this only works when the year of record falls between 1678 and 2262. Since you have years 2262-2300 in your dataset, this is a feature - xray is failing gracefully.

There are a few current open discussions on this behavior, which is an issue higher up the python chain with numpy:

1. time decoding error with "days since"
2. Fix datetime decoding when time units are 'days since 0000-01-01 00:00:00'
3. ocefpaf - Loading non-standard dates with cf_units
4. numpy - Non-standard Calendar Support

For now, a very simple hack would be to re-compute your time units so that they're re-based, say, with units 'days since 1700-01-01 00:00:00'. That way all of them would fit within the permissible range to use the decoding routine built into xray. You could simply pass the decode_cf=False flag when you open the dataset, modify the non-decoded time array and units, then run xray.decode_cf() on the modified dataset.

[jsbj]

Thanks Daniel, that workaround totally works.

-j

2015-08-14 15:26 GMT-04:00 Daniel Rothenberg [email protected]:

Hi @jsbj https://github.com/jsbj,

The fancy indexing notation you're trying to use only works when xray
successfully decodes the time dimension. As discussed in the documentation
here
http://xray.readthedocs.org/en/stable/time-series.html#creating-datetime64-data,
this only works when the year of record falls between 1678 and 2262. Since
you have years 2262-2300 in your dataset, this is a feature - xray is
failing gracefully.

There are a few current open discussions on this behavior, which is an
issue higher up the python chain with numpy:

1. time decoding error with "days since"
   time decoding error with "days since"  #521
2. Fix datetime decoding when time units are 'days since 0000-01-01
   00:00:00' Fix datetime decoding when time units are 'days since 0000-01-01 00:00:00' #522
3. ocefpaf - Loading non-standard dates with cf_units
   https://ocefpaf.github.io/python4oceanographers/blog/2015/08/10/cf_units_and_time/
4. numpy - Non-standard Calendar Support
   Non-standard Calendar Support numpy/numpy#6207

For now, a very simple hack would be to re-compute your time units so that
they're re-based, say, with units 'days since 1700-01-01 00:00:00'. That
way all of them would fit within the permissible range to use the decoding
routine built into xray. You could simply pass the decode_cf=False flag
when you open the dataset, modify the non-decoded time array and units,
then run xray.decode_cf() on the modified dataset.

—
Reply to this email directly or view it on GitHub
#531 (comment).

[jhamman]

@jsbj - glad to hear you were able to work around this. Extended calendar support in numpy, pandas, and xray is something we're all rooting for.

I'm going to close this as a wontfix since we are working on this via the numpy and pandas channels.