[Bug]: OutOfBoundsDatetime: Out of bounds nanosecond timestamp: 2262-05-01 00:00:00

[pochedls]

What happened?

In opening a dataset with time units months since 1800-01-01 I am getting the following error: OutOfBoundsDatetime: Out of bounds nanosecond timestamp: 2262-05-01 00:00:00.

What did you expect to happen?

Ideally, xcdat could handle this situation and open the dataset.

Minimal Complete Verifiable Example

import xcdat

fn = '/p/user_pub/climate_work/pochedley1/cmip6_msu/spliced/ttt_Amon_MRI-ESM2-0_historical-ssp585_r1i1p1f1_gn_185001-230012.nc'
ds = xcdat.open_dataset(fn)

Relevant log output

---------------------------------------------------------------------------                                                                                          
OutOfBoundsDatetime                       Traceback (most recent call last)
Input In [65], in <cell line: 1>()
----> 1 ds = xcdat.open_dataset(fn)

File ~/code/xcdat/xcdat/dataset.py:89, in open_dataset(path, data_var, add_bounds, decode_times, center_times, lon_orient, **kwargs)
     87     ds = xr.open_dataset(path, decode_times=False, **kwargs)
     88     # attempt to decode non-cf-compliant time axis
---> 89     ds = decode_non_cf_time(ds)
     90 else:
     91     ds = xr.open_dataset(path, decode_times=True, **kwargs)

File ~/code/xcdat/xcdat/dataset.py:320, in decode_non_cf_time(dataset)
    316     return ds
    318 ref_date = pd.to_datetime(ref_date)
--> 320 data = [ref_date + pd.DateOffset(**{units: offset}) for offset in time.data]
    321 decoded_time = xr.DataArray(
    322     name=time.name,
    323     data=data,
   (...)
    326     attrs=time.attrs,
    327 )
    328 decoded_time.encoding = {
    329     "source": ds.encoding.get("source", "None"),
    330     "dtype": time.dtype,
   (...)
    333     "calendar": time.attrs.get("calendar", "none"),
    334 }

File ~/code/xcdat/xcdat/dataset.py:320, in <listcomp>(.0)
    316     return ds
    318 ref_date = pd.to_datetime(ref_date)
--> 320 data = [ref_date + pd.DateOffset(**{units: offset}) for offset in time.data]
    321 decoded_time = xr.DataArray(
    322     name=time.name,
    323     data=data,
   (...)
    326     attrs=time.attrs,
    327 )
    328 decoded_time.encoding = {
    329     "source": ds.encoding.get("source", "None"),
    330     "dtype": time.dtype,
   (...)
    333     "calendar": time.attrs.get("calendar", "none"),
    334 }

File ~/bin/anaconda3/envs/xcdat_dev/lib/python3.9/site-packages/pandas/_libs/tslibs/offsets.pyx:444, in pandas._libs.tslibs.offsets.BaseOffset.__add__()

File ~/bin/anaconda3/envs/xcdat_dev/lib/python3.9/site-packages/pandas/_libs/tslibs/offsets.pyx:450, in pandas._libs.tslibs.offsets.BaseOffset.__add__()

File ~/bin/anaconda3/envs/xcdat_dev/lib/python3.9/site-packages/pandas/_libs/tslibs/offsets.pyx:180, in pandas._libs.tslibs.offsets.apply_wraps.wrapper()

File ~/bin/anaconda3/envs/xcdat_dev/lib/python3.9/site-packages/pandas/_libs/tslibs/offsets.pyx:1092, in pandas._libs.tslibs.offsets.RelativeDeltaOffset._apply()

File ~/bin/anaconda3/envs/xcdat_dev/lib/python3.9/site-packages/pandas/_libs/tslibs/timestamps.pyx:1399, in pandas._libs.tslibs.timestamps.Timestamp.__new__()

File ~/bin/anaconda3/envs/xcdat_dev/lib/python3.9/site-packages/pandas/_libs/tslibs/conversion.pyx:436, in pandas._libs.tslibs.conversion.convert_to_tsobject()

File ~/bin/anaconda3/envs/xcdat_dev/lib/python3.9/site-packages/pandas/_libs/tslibs/conversion.pyx:517, in pandas._libs.tslibs.conversion.convert_datetime_to_tsobject()

File ~/bin/anaconda3/envs/xcdat_dev/lib/python3.9/site-packages/pandas/_libs/tslibs/np_datetime.pyx:120, in pandas._libs.tslibs.np_datetime.check_dts_bounds()

OutOfBoundsDatetime: Out of bounds nanosecond timestamp: 2262-05-01 00:00:00

Anything else we need to know?

This seems to be related to a known limitation of pandas. Maybe using errors="coerce" could help?

Environment

main branch

[pochedls]

Addendum that this has apparently been addressed in xarray (but not for non-CF time, which we are supporting in this case).

[durack1]

@pochedls @tomvothecoder was this to be fixed in 0.3.1? It seems I have hit an almost identical issue:

OutOfBoundsDatetime: Out of bounds nanosecond timestamp: 2315-07-01 00:00:00

The offending file time dimension looks like:

...
        float time(time) ;
                time:standard_name = "time" ;
                time:long_name = "time" ;
                time:units = "months since 1955-01-01 00:00:00" ;
                time:axis = "T" ;
                time:climatology = "climatology_bounds" ;
...

In [8]: xc.__version__
Out[8]: '0.3.0'

Is 0.3.1 expected to be available on the conda-forge channel imminently?

[tomvothecoder]

@durack1 This is fixed in #283, which is merged on the latest main.

We are addressing a few more bugs so 0.3.1 is not released yet, but should be pretty soon.
https://github.com/orgs/xCDAT/projects/3/views/2?query=is%3Aopen+sort%3Aupdated-desc