Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

BUG: nonexistent Timestamp pre-summer/winter DST w/dateutil timezone #31155

Merged
merged 26 commits into from
Jan 24, 2020
Merged
Show file tree
Hide file tree
Changes from 13 commits
Commits
Show all changes
26 commits
Select commit Hold shift + click to select a range
651a55f
BUG: nonexistent Timestamp pre-summer/winter DST change with dateutil…
AlexKirko Jan 20, 2020
c9a87bd
switch from scientific number notation
AlexKirko Jan 20, 2020
ca34eed
move back to scientific notation to reset tests
AlexKirko Jan 20, 2020
65b3bb8
move away from scientific notation to reset tests
AlexKirko Jan 20, 2020
1eb9500
add casts to force correct division
AlexKirko Jan 20, 2020
2f3850e
switch to dateutil implementation
AlexKirko Jan 20, 2020
6c87f1b
TST: expand test for debugging
AlexKirko Jan 20, 2020
43f6645
TST: fix test expansion
AlexKirko Jan 20, 2020
b1defde
go back to previous implementation
AlexKirko Jan 20, 2020
e46c774
hopefully a more robust implementation
AlexKirko Jan 21, 2020
4f8b490
add rounding
AlexKirko Jan 21, 2020
f8dfb36
TST: remove exception from another test
AlexKirko Jan 21, 2020
3ad3212
add more to the test
AlexKirko Jan 21, 2020
2174ca0
try removing rounding, see if it breaks
AlexKirko Jan 21, 2020
0cf53c1
add value stability and skip condition to the test
AlexKirko Jan 21, 2020
3f79c3d
TST: add xfails for dateutil version < 2.7.0
AlexKirko Jan 21, 2020
1c147d3
TST: remove duplicate test
AlexKirko Jan 21, 2020
f03e7c9
TST: remove packaging import to find the error
AlexKirko Jan 21, 2020
c88e354
TST: switch to LooseVersion for dateutil version check
AlexKirko Jan 21, 2020
109a285
TST: use get_distribution to correct mypy error
AlexKirko Jan 21, 2020
8e754f7
TST: fix value stability test
AlexKirko Jan 22, 2020
894ed16
Revert "TST: fix value stability test"
AlexKirko Jan 22, 2020
614176f
Revert Revert "TST: fix value stability test"
AlexKirko Jan 22, 2020
56a3e71
TST: move to compat._optional._get_version
AlexKirko Jan 22, 2020
89a5c01
Merge branch 'master' into fix-nonexistent-time
AlexKirko Jan 22, 2020
9d21a81
restart checks
AlexKirko Jan 22, 2020
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
3 changes: 2 additions & 1 deletion doc/source/whatsnew/v1.1.0.rst
Original file line number Diff line number Diff line change
Expand Up @@ -59,9 +59,10 @@ Categorical

Datetimelike
^^^^^^^^^^^^

- Bug in :class:`Timestamp` where constructing :class:`Timestamp` from ambiguous epoch time and calling constructor again changed :meth:`Timestamp.value` property (:issue:`24329`)
- :meth:`DatetimeArray.searchsorted`, :meth:`TimedeltaArray.searchsorted`, :meth:`PeriodArray.searchsorted` not recognizing non-pandas scalars and incorrectly raising ``ValueError`` instead of ``TypeError`` (:issue:`30950`)
-
- Bug in :class:`Timestamp` where constructing :class:`Timestamp` with dateutil timezone less than 128 nanoseconds before daylight saving time switch from winter to summer would result in nonexistent time (:issue:`31043`)

Timedelta
^^^^^^^^^
Expand Down
2 changes: 1 addition & 1 deletion pandas/_libs/tslibs/nattype.pyx
Original file line number Diff line number Diff line change
Expand Up @@ -278,7 +278,7 @@ cdef class _NaT(datetime):

def total_seconds(self):
"""
Total duration of timedelta in seconds (to ns precision).
Total duration of timedelta in seconds (to microsecond precision).
"""
# GH#10939
return np.nan
Expand Down
6 changes: 4 additions & 2 deletions pandas/_libs/tslibs/timedeltas.pyx
Original file line number Diff line number Diff line change
Expand Up @@ -861,9 +861,11 @@ cdef class _Timedelta(timedelta):

def total_seconds(self):
"""
Total duration of timedelta in seconds (to ns precision).
Total duration of timedelta in seconds (to microsecond precision).
"""
return self.value / 1e9
# GH 31043
# Microseconds precision to avoid confusing tzinfo.utcoffset
return np.round((self.value - self.value % 1000) / 1e9, 6)

def view(self, dtype):
"""
Expand Down
7 changes: 1 addition & 6 deletions pandas/tests/indexes/datetimes/test_timezones.py
Original file line number Diff line number Diff line change
Expand Up @@ -581,12 +581,7 @@ def test_dti_construction_ambiguous_endpoint(self, tz):
["US/Pacific", "shift_forward", "2019-03-10 03:00"],
["dateutil/US/Pacific", "shift_forward", "2019-03-10 03:00"],
["US/Pacific", "shift_backward", "2019-03-10 01:00"],
pytest.param(
"dateutil/US/Pacific",
"shift_backward",
"2019-03-10 01:00",
marks=pytest.mark.xfail(reason="GH 24329"),
),
["dateutil/US/Pacific", "shift_backward", "2019-03-10 01:00"],
["US/Pacific", timedelta(hours=1), "2019-03-10 03:00"],
],
)
Expand Down
14 changes: 14 additions & 0 deletions pandas/tests/scalar/timestamp/test_timestamp.py
Original file line number Diff line number Diff line change
Expand Up @@ -1092,3 +1092,17 @@ def test_constructor_ambigous_dst():
expected = ts.value
result = Timestamp(ts).value
assert result == expected


def test_constructor_before_dst_switch():
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Can we test the example in the issue here? (i.e. that Timestamp(Timestamp(epoch_time, tz=..)) doesn't change value)

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@mroeschke Sure!
But the value breaks only 1 nanosecond before the switch. Currently, I'm struggling to make everything else work on every azure pipeline (see comment below).

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@mroeschke
Added to the test. Pinging you on change.

# GH 31043
# Make sure that calling Timestamp constructor
# on time just before DST switch doesn't lead to
# nonexistent time
epoch = 1552211999999999872
ts = Timestamp(epoch, tz="dateutil/US/Pacific")
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Forgot to mention earlier: you should probably use dateutil/America/Los_Angeles, as that is the canonical name for this zone. The US/... zones are symlinks for backwards compatibility.

delta = ts.replace(tzinfo=None) - datetime.utcfromtimestamp(0)
assert 1552183199.99 < delta.total_seconds() < 1552183200
expected = timedelta(seconds=0)
result = ts.tz.dst(ts)
assert result == expected
AlexKirko marked this conversation as resolved.
Show resolved Hide resolved