Last of the timezones funcs #17669

jbrockmendel · 2017-09-25T18:12:51Z

Moves tools.datetimes._infer_tzinfo to tslibs.timezones._infer_tzinfo.

Refactor a large timezone-specific chunk of tslib.tz_localize_to_utc to tslibs.timezones._infer_dst

The new func timezones._infer_dst is taken out of tslib.tz_localize_to_utc. It is very nearly a cut/paste. The only change is removed a couple calls to Timestamp that are used for rendering an exception message, used np.datetime64 in its place.

closes #xxxx
tests added / passed
passes git diff upstream/master -u -- "*.py" | flake8 --diff
whatsnew entry

refactor a large timezone-specific chunk of tslib.tz_localize_to_utc to tslibs.timezones._infer_dst

jreback · 2017-09-25T22:29:38Z

pandas/_libs/tslibs/timezones.pyx

+                                 ndarray[int64_t] result_b):
+    cdef:
+        Py_ssize_t n = len(vals)
+        ndarray[int64_t] dst_hours


make sure u r typing s the original
there are lots of issues here

The typings should be identical. There are a couple of variables that do not have type declarations; those can be added.

jreback · 2017-09-25T22:30:28Z

pandas/core/tools/datetimes.py

-def _infer_tzinfo(start, end):
-    def _infer(a, b):
-        tz = a.tzinfo
-        if b and b.tzinfo:


where is this actually used?

Outside of tests, its used once in indexes.datetimes

jreback · 2017-09-25T22:30:49Z

pandas/_libs/tslibs/timezones.pyx

@@ -275,3 +275,84 @@ cdef object get_dst_info(object tz):
        dst_cache[cache_key] = (trans, deltas, typ)

    return dst_cache[cache_key]
+
+
+def _infer_tzinfo(start, end):


de-privatize these

jreback · 2017-09-25T22:31:10Z

pandas/_libs/tslibs/timezones.pyx

+
+def _infer_tzinfo(start, end):
+    def _infer(a, b):
+        tz = a.tzinfo


this should be an outside function

If we're going to bother changing it, might as well get rid of _infer. It's only actually relevant in one case (of four)

jreback · 2017-09-25T23:11:23Z

pandas/_libs/tslib.pyx

@@ -105,7 +105,7 @@ from tslibs.timezones cimport (
    is_utc, is_tzlocal, is_fixed_offset,
    treat_tz_as_dateutil, treat_tz_as_pytz,
    get_timezone, get_utcoffset, maybe_get_tz,
-    get_dst_info
+    get_dst_info, _infer_dst


let's give this a more descriptive name: infer_dst_transitions

jreback · 2017-09-25T23:11:36Z

pandas/_libs/tslibs/timezones.pyx

+
+def _infer_tzinfo(start, end):
+    def _infer(a, b):
+        tz = a.tzinfo


jreback · 2017-09-25T23:16:10Z

pandas/_libs/tslibs/timezones.pyx

+    return tz
+
+
+cdef ndarray[int64_t] _infer_dst(ndarray[int64_t] vals,


you are removing part of this function. I think its better to move almost all of it to timezones or leave it. (ok with latter for now, maybe do former at some point). Otherwise you end up splitting the logic / comments in 2 places.

I think this requires a bit more thought here.

This is the appropriate block to separate out. It is exclusively focused on getting DST info for use in the original function. The rest of the function is thematically more related to _TSObject conversion.

In particular, pandas_datetimestruct doesn't belong anywhere near timezones.

codecov · 2017-09-26T00:21:54Z

Codecov Report

Merging #17669 into master will decrease coverage by 0.01%.
The diff coverage is 100%.

@@            Coverage Diff             @@
##           master   #17669      +/-   ##
==========================================
- Coverage   91.25%   91.23%   -0.02%     
==========================================
  Files         163      163              
  Lines       49808    49798      -10     
==========================================
- Hits        45454    45435      -19     
- Misses       4354     4363       +9

Flag	Coverage Δ
#multiple	`89.03% <100%> (-0.01%)`	⬇️
#single	`40.31% <100%> (-0.07%)`	⬇️

Impacted Files	Coverage Δ
pandas/core/tools/datetimes.py	`84.74% <100%> (-0.51%)`	⬇️
pandas/io/gbq.py	`25% <0%> (-58.34%)`	⬇️
pandas/core/frame.py	`97.73% <0%> (-0.1%)`	⬇️
pandas/core/indexes/base.py	`96.29% <0%> (ø)`	⬆️

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update e0fe5cc...355cbe8. Read the comment docs.

codecov · 2017-09-26T00:21:55Z

Codecov Report

Merging #17669 into master will increase coverage by <.01%.
The diff coverage is 100%.

@@            Coverage Diff             @@
##           master   #17669      +/-   ##
==========================================
+ Coverage   91.24%   91.25%   +<.01%     
==========================================
  Files         163      163              
  Lines       49766    49770       +4     
==========================================
+ Hits        45411    45417       +6     
+ Misses       4355     4353       -2

Flag	Coverage Δ
#multiple	`89.04% <100%> (+0.02%)`	⬆️
#single	`40.33% <100%> (-0.07%)`	⬇️

Impacted Files	Coverage Δ
pandas/core/tools/datetimes.py	`83.03% <ø> (-0.77%)`	⬇️
pandas/core/indexes/datetimes.py	`95.53% <100%> (ø)`	⬆️
pandas/io/gbq.py	`25% <0%> (-58.34%)`	⬇️
pandas/core/frame.py	`97.73% <0%> (-0.1%)`	⬇️
pandas/core/indexes/interval.py	`92.85% <0%> (ø)`	⬆️
pandas/core/sparse/array.py	`91.58% <0%> (ø)`	⬆️
pandas/compat/numpy/function.py	`93.33% <0%> (+0.2%)`	⬆️
pandas/plotting/_converter.py	`65.2% <0%> (+1.81%)`	⬆️

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 074b485...ce701e5. Read the comment docs.

…libs-timezones7

jreback · 2017-09-27T10:21:56Z

pandas/_libs/tslib.pyx

-        both_eq = result_a == result_b
-        trans_idx = np.squeeze(np.nonzero(np.logical_and(both_nat, ~both_eq)))
-        if trans_idx.size == 1:
-            stamp = Timestamp(vals[trans_idx])


revert this routine.

jreback

just do the infer_tzinfo for now

…libs-timezones7

jreback · 2017-09-28T10:34:37Z

ok rebase, ping on green.

…libs-timezones7

jreback · 2017-09-29T10:16:41Z

thanks

* 'master' of github.com:pandas-dev/pandas: (188 commits) Separate out _convert_datetime_to_tsobject (pandas-dev#17715) DOC: remove whatsnew note for xref pandas-dev#17131 BUG: Regression in .loc accepting a boolean Index as an indexer (pandas-dev#17738) DEPR: Deprecate cdate_range and merge into bdate_range (pandas-dev#17691) CLN: replace %s syntax with .format in pandas.core: categorical, common, config, config_init (pandas-dev#17735) Fixed the memory usage explanation of categorical in gotchas from O(nm) to O(n+m) (pandas-dev#17736) TST: add backward compat for offset testing for pickles (pandas-dev#17733) remove unused time conversion funcs (pandas-dev#17711) DEPR: Deprecate convert parameter in take (pandas-dev#17352) BUG:Time Grouper bug fix when applied for list groupers (pandas-dev#17587) BUG: Fix some PeriodIndex resampling issues (pandas-dev#16153) BUG: Fix unexpected sort in groupby (pandas-dev#17621) DOC: Fixed typo in documentation for 'pandas.DataFrame.replace' (pandas-dev#17731) BUG: Fix series rename called with str altering name rather index (GH17407) (pandas-dev#17654) DOC: Add examples for MultiIndex.get_locs + cleanups (pandas-dev#17675) Doc improvements for IntervalIndex and Interval (pandas-dev#17714) BUG: DataFrame sort_values and multiple "by" columns fails to order NaT correctly Last of the timezones funcs (pandas-dev#17669) Add missing file to _pyxfiles, delete commented-out (pandas-dev#17712) update imports of DateParseError, remove unused imports from tslib (pandas-dev#17713) ...

jbrockmendel added 3 commits September 25, 2017 11:05

move infer_tzinfo from tools.datetime to tslibs.timezones

1932996

refactor a large timezone-specific chunk of tslib.tz_localize_to_utc to tslibs.timezones._infer_dst

whitespace fixup

6ccbfb2

fixup add missing cimport

adf9099

jreback requested changes Sep 25, 2017

View reviewed changes

de-privatize infer_tzinfo, update imports

fcd26c1

jreback requested changes Sep 25, 2017

View reviewed changes

jreback added the Timezones Timezone data dtype label Sep 25, 2017

rename _infer_dst --> infer_dst_transitions per reviewer request

355cbe8

jbrockmendel added 4 commits September 25, 2017 17:24

Remove nested _infer func

e17089c

flake8 fixup

b82396d

Merge branch 'master' of https://github.com/pandas-dev/pandas into ts…

e6096c5

…libs-timezones7

fixup remove unused import

926bb73

jreback reviewed Sep 27, 2017

View reviewed changes

jreback requested changes Sep 27, 2017

View reviewed changes

jbrockmendel added 3 commits September 27, 2017 20:14

Merge branch 'master' of https://github.com/pandas-dev/pandas into ts…

2afc7b2

…libs-timezones7

Merge branch 'master' of https://github.com/pandas-dev/pandas into ts…

8f4b368

…libs-timezones7

rever infer_dst_transition

7b4c9b2

jreback added this to the 0.21.0 milestone Sep 28, 2017

jbrockmendel added 2 commits September 28, 2017 08:06

Merge branch 'master' of https://github.com/pandas-dev/pandas into ts…

0248f53

…libs-timezones7

dummy commit to force CI

ce701e5

jreback approved these changes Sep 29, 2017

View reviewed changes

jreback merged commit 54f6648 into pandas-dev:master Sep 29, 2017

jbrockmendel deleted the tslibs-timezones7 branch October 30, 2017 16:23

alanbato pushed a commit to alanbato/pandas that referenced this pull request Nov 10, 2017

Last of the timezones funcs (pandas-dev#17669)

55516a8

No-Stream pushed a commit to No-Stream/pandas that referenced this pull request Nov 28, 2017

Last of the timezones funcs (pandas-dev#17669)

75fc2f9

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Last of the timezones funcs #17669

Last of the timezones funcs #17669

jbrockmendel commented Sep 25, 2017

jreback Sep 25, 2017

jbrockmendel Sep 26, 2017

jreback Sep 25, 2017

jbrockmendel Sep 25, 2017

jreback Sep 25, 2017

jbrockmendel Sep 25, 2017

jreback Sep 25, 2017

jbrockmendel Sep 25, 2017

jreback Sep 25, 2017

jreback Sep 25, 2017

jbrockmendel Sep 26, 2017

jreback Sep 25, 2017

jreback Sep 25, 2017

jbrockmendel Sep 26, 2017

codecov bot commented Sep 26, 2017

codecov bot commented Sep 26, 2017 •

edited

Loading

jreback Sep 27, 2017

jreback left a comment

jreback commented Sep 28, 2017

jreback commented Sep 29, 2017

		return tz


		cdef ndarray[int64_t] _infer_dst(ndarray[int64_t] vals,

Last of the timezones funcs #17669

Last of the timezones funcs #17669

Conversation

jbrockmendel commented Sep 25, 2017

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

codecov bot commented Sep 26, 2017

Codecov Report

codecov bot commented Sep 26, 2017 • edited Loading

Codecov Report

Choose a reason for hiding this comment

jreback left a comment

Choose a reason for hiding this comment

jreback commented Sep 28, 2017

jreback commented Sep 29, 2017

codecov bot commented Sep 26, 2017 •

edited

Loading