v1.1.8 'days since 0000-01-01 00:00:00' ValueError: year is out of range

[frodre]

I got a strange error when working with some data today and it was related to the time variable's units.

In [22]: time.calendar
Out[22]: u'noleap'

In [23]: time.units
Out[23]: u'days since 0000-01-01 00:00:00'

When code attempted to translate the time data into datetime objects I would get the following error:

  File "netcdf_year_value_error.py", line 10, in <module>
    t_yrs = ncf.num2date(time[:], units=time.units, calendar=time.calendar)
  File "netCDF4/_netCDF4.pyx", line 4522, in netCDF4._netCDF4.num2date (netCDF4/_netCDF4.c:49796)
  File "netCDF4/_netCDF4.pyx", line 4337, in netCDF4._netCDF4._dateparse (netCDF4/_netCDF4.c:47646)
ValueError: year is out of range

After digging around a bit, it was apparent that the limit for gregorian calendar (which is what 'noleap' translates to?) is the year 0001 CE. However, I was looking at some documentation in netcdftime/netcdftime.py and this section (line 612)

"udunits implements the mixed Gregorian/Julian calendar system, as
followed in England, in which dates prior to 1582-10-15 are assumed to use
the Julian calendar. Other software cannot be relied upon to handle the
change of calendar in the same way, so for robustness it is recommended
that the reference date be later than 1582. If earlier dates must be used,
it should be noted that udunits treats 0 AD as identical to 1 AD."

This makes it seem like 'days since 0000-01-01 00:00:00' should work and be interpreted as the year 0001. I also have a colleague who was running the same code (not sure of the version for his netcdf4 library) without the ValueError.

I switched the units to use the year 0001 and it ended up working fine, but the data is now of course pushed 1 year forward. I figured I'd throw this on here just incase it is a bug, and not an intended feature.

I'm using Python v2.7.10 with the Anaconda distr. v2.2.0 and netCDF4 v1.1.8.

• frodre

[jswhit]

There is no year 0 in the Gregorian or Julian calendars. I don't think we should try to support this by treating year 0 the same as year 1 as udunits does.

https://en.wikipedia.org/wiki/0_%28year%29

[frodre]

Yeah, that makes sense. Perhaps making a more descriptive ValueError for this would be useful? There are datasets floating around with this time units convention floating around...

[rabernat]

I just encountered this error myself and then found it was already documented here. 'days since 0000-01-01 00:00:00' is the unit for time in the CCSM POP ocean model. It will be problematic if it is not supported by netCDF4, since there is lots of pre-existing output that uses this convention.

What would be the best work-around, short of modifying petabytes of existing model output?

[frodre]

I've just been using a method where I shift the year to something allowable, use num2date, and then shift it back. Saves me from having to alter the netcdf files....

    try:
        time = num2date(time_var[:], units=time_var.units,
                            calendar=time_var.calendar)
        return time.tolist()
    except ValueError:
        # num2date needs calendar year start >= 0001 C.E. (bug submitted
        # to unidata about this
        tunits = time_var.units
        since_yr_idx = tunits.index('since ') + 6
        year = int(tunits[since_yr_idx:since_yr_idx+4])
        year_diff = year - 0001

        new_units = tunits[:since_yr_idx] + '0001-01-01 00:00:00'
        time = num2date(time_var[:], new_units, calendar=time_var.calendar)
        return [datetime(d.year + year_diff, d.month, d.day,
                         d.hour, d.minute, d.second)
                for d in time]

[ocefpaf]

I agree with @jswhit that netcdf4-python should not support this. But some CDMs, like iris, can handle this for you. Iris uses cf_units which obey UDUNITS rules:

import cf_units
from datetime import datetime

units = "days since 0000-01-01 00:00:00"

cf_units.date2num(datetime(1, 1, 1), units, cf_units.CALENDAR_NO_LEAP)
365.0

cf_units.num2date(365, units, cf_units.CALENDAR_STANDARD)
-1-12-31 00:00:00  # (!)

cf_units.num2date(366, units, cf_units.CALENDAR_STANDARD)
1-01-01 00:00:00

A little odd, but works!

[jswhit]

pull request #447 adds a more informative error message if year 0 (or negative year) used in a time units string.

[davidhassell]

Hi,

I see that zero is not allowed for other calendars, too:

In [9]: netCDF4.__version__
Out[9]: '1.2.0'
In [10]: netCDF4.netcdftime.utime('days since 0-1-1', '360_day')
<snip>
ValueError: zero not allowed as a reference year, does not exist in Julian or Gregorian calendars

The CF conventions says "Year 0 may be a valid year in non-real-world calendars, and therefore cannot be used to signal climatological time in such cases.", so it ought to be ok in the non-real-world cases. What do you think?

All the best,

David

[jswhit]

I agree. Pull request #470 fixes this (allows non-positive reference years in non-real-world calendars).

[jswhit]

Pull request #470 has been merged.

[davidhassell]

That's great - thanks.