fixing open-ended adjustements error #216
Conversation
While trying to prescribe an open-ended adjustments, I noticed that it currently causes an error. When we read the start end end times of adjustments, they receive a time zone info due to their ISO format. The AWS xarray dataset does not have a time zone info (because of an [xarray limitation](pydata/xarray#3291)). So the timezone info is removed from the adjustments time bounds (l.183-184). What was missing is that when start or end date of adjustments are blank (meaning open-start, open-ended bounds), we use a timestamp (then time-zone-naive) from the AWS dataset, and that it then causes an error later on when trying to remove the time-zone info from these same time-zone-naive bounds.
switching copies to deep copies
Co-authored-by: Mads Christian Lund <[email protected]>
| adj_info[['t0','t1']] = adj_info[['t0','t1']].astype(object) | ||
| adj_info.loc[adj_info.t1.isnull()|(adj_info.t1==''), "t1"] = None | ||
| adj_info.loc[adj_info.t0.isnull()|(adj_info.t0==''), "t0"] = None |
There was a problem hiding this comment.
t0 and t1 are already checked in line 219:222 except for the empty string and np.nan
Consider instead
if isinstance(t0, str) and t0 != '':
t0 = pd.to_datetime(t0, utc=True).tz_localize(None)
else:
t0 = None
if isinstance(t1, str) and t1 != '':
t1 = pd.to_datetime(t1, utc=True).tz_localize(None)
else:
t1 = NoneThere was a problem hiding this comment.
Not a big fan of if/else in for loop. I'd rather prepare the adj_info table before looping through it.
Now I see that I have used part of your suggestion l.219-222 so I admit it is currently a bit hybrid.
| adj_info.t1 = pd.to_datetime(adj_info.t1).dt.tz_localize(None) | ||
|
|
||
| # making sure that t0 and t1 columns are object dtype then replaceing nan with None | ||
| adj_info[['t0','t1']] = adj_info[['t0','t1']].astype(object) |
There was a problem hiding this comment.
I don't understand why you cast the type to object. I suppose the dynamically inferred types are either
strif all the values in the columns are stringsfloatif all the values in the columns inferred as nanobjectif the values in the columns have different types
There was a problem hiding this comment.
I guess that, along with your next comment, it is a matter of personal preference:
I prefer replacing all the missing dates by None in two single-line vectorized call (l.181-182) rather than having ifs in a for loop. But before I do that, I need to make sure that the t0 and t1 columns can accommodate None. If a t0 or t1 column has a float type and I try to replace the values by None pandas actually coerces those None into their float equivalent: np.nan. This causes problem later as slice(np.nan, np.nan) fails. When t0 and t1 are of object type, then replacing some of their values by None will keep the None as NoneType.
While trying to prescribe an open-ended adjustments, I noticed that it currently causes an error.
When we read the start end end times of adjustments, they receive a time zone info due to their ISO format. The AWS xarray dataset does not have a time zone info (because of an xarray limitation). So the timezone info is removed from the adjustments time bounds (l.183-184).
What was missing is that when start or end date of adjustments are blank (meaning open-start, open-ended bounds), we use a timestamp (then time-zone-naive) from the AWS dataset, and that it then causes an error later on when trying to remove the time-zone info from these same time-zone-naive bounds.