Fix Index mul-like ops with timedelta scalars #19333

jbrockmendel · 2018-01-21T21:46:55Z

Fixes the following current behavior for each of the numeric index classes and each of the basic timedelta-like scalars:

idx = pd.Index(range(3))
td = pd.Timedelta(days=1)

>>> idx * td
TypeError: can only perform ops with timedelta like values
>>> td * idx
TypeError: can only perform ops with timedelta like values
>>> td / idx
TypeError: can only perform ops with timedelta like values
>>> td // idx
TypeError: can only perform ops with timedelta like values

closes #xxxx
tests added / passed
passes git diff upstream/master -u -- "*.py" | flake8 --diff
whatsnew entry

…x_mul_tdi

jreback · 2018-01-21T22:11:46Z

doc/source/whatsnew/v0.23.0.txt

@@ -428,7 +428,9 @@ Conversion
 - Bug in localization of a naive, datetime string in a ``Series`` constructor with a ``datetime64[ns, tz]`` dtype (:issue:`174151`)
 - :func:`Timestamp.replace` will now handle Daylight Savings transitions gracefully (:issue:`18319`)
 - Bug in :class:`Index` multiplication and division methods where operating with a ``Series`` would return an ``Index`` object instead of a ``Series`` object (:issue:`19042`)
-
+- Multiplication of :class:`TimedeltaIndex` by ``TimedeltaIndex`` will now raise ``TypeError`` instead of raising ``ValueError`` in cases of length mis-match (:issue:`????`)


when you don't have an issue number, use the PR number

jreback · 2018-01-21T22:13:33Z

pandas/core/indexes/base.py

@@ -3864,7 +3864,21 @@ def dropna(self, how='any'):
        return self._shallow_copy()

    def _evaluate_with_timedelta_like(self, other, op, opstr, reversed=False):
-        raise TypeError("can only perform ops with timedelta like values")
+        # Timedelta knows how to operate with np.array, so dispatch to that


shouldn't this be left alone, and rather define this in TimedeltaIndex?

This for for numeric-dtyped indexes. e.g. pd.Index(range(3)) * pd.Timedelta(days=1) goes through this path.

why doesn't this just raise NotImplementedError? which then Timedelta would handle? or is that too recursive a path?

For one thing b/c there’s pytimedelta and timedelta64 that need to be handled.

jreback · 2018-01-21T22:33:47Z

I just updated the whatsnew, to be more organized, so rebase again.

jbrockmendel · 2018-01-21T22:38:13Z

OK. Should I put the multiplication/division stuff in Numeric instead of Conversion? A few days back you suggested an Ops section, but I wasn't clear on what the scope of that would be.

…x_mul_tdi

jreback · 2018-01-21T22:42:10Z

I would put this in datetimelike, numeric is for well, numeric types! this is sort of an arbitrary split but makes each section shorter

jreback · 2018-01-21T22:42:40Z

doc/source/whatsnew/v0.23.0.txt

@@ -446,6 +447,9 @@ Numeric
 - Bug in :class:`Index` multiplication and division methods where operating with a ``Series`` would return an ``Index`` object instead of a ``Series`` object (:issue:`19042`)
 - Bug in the :class:`DataFrame` constructor in which data containing very large positive or very large negative numbers was causing ``OverflowError`` (:issue:`18584`)
 - Bug in :class:`Index` constructor with ``dtype='uint64'`` where int-like floats were not coerced to :class:`UInt64Index` (:issue:`18400`)
+- Bug in :class:`Index` multiplication and division methods where operating with a ``Series`` would return an ``Index`` object instead of a ``Series`` object (:issue:`19042`)
+- Multiplication of :class:`TimedeltaIndex` by ``TimedeltaIndex`` will now raise ``TypeError`` instead of raising ``ValueError`` in cases of length mis-match (:issue`19333`)


1st one ok, other 2 in datetimelike section

jreback · 2018-01-22T11:17:06Z

pandas/tests/indexes/test_numeric.py

@@ -26,6 +26,36 @@ def full_like(array, value):
    return ret


+class TestIndexArithmetic(object):
+    @pytest.mark.parametrize('index', [pd.Int64Index(range(1, 11)),


can you add the issue number as a comment (or PR number if no issue)

jreback · 2018-01-22T11:17:34Z

pandas/tests/indexes/test_numeric.py

+class TestIndexArithmetic(object):
+    @pytest.mark.parametrize('index', [pd.Int64Index(range(1, 11)),
+                                       pd.UInt64Index(range(1, 11)),
+                                       pd.Float64Index(range(1, 11)),


is this how we are importing in this module, e.g. pd.*? usually we don't do this (its ok whichever, just want to be consistent)

Looks like I'm being inconsistent here, will change.

jreback · 2018-01-22T11:19:03Z

pandas/tests/indexes/test_numeric.py

+        expected = pd.TimedeltaIndex(['1 Day', '12 Hours'])
+
+        result = scalar_td / index
+        tm.assert_index_equal(result, expected)


can you also test that the commute raises here

jreback · 2018-01-22T11:19:40Z

pandas/tests/indexes/timedeltas/test_arithmetic.py

@@ -297,7 +297,7 @@ def test_dti_mul_dti_raises(self):

    def test_dti_mul_too_short_raises(self):
        idx = self._holder(np.arange(5, dtype='int64'))
-        with pytest.raises(ValueError):


why did this change?

Because the problem is now caught in TimedeltaIndex when checking what it is operating against, whereas before the problem was caught in numpy when trying to operate on differently-sized arrays.

jreback · 2018-01-22T11:21:45Z

pandas/tests/indexes/test_numeric.py

+                                       pd.UInt64Index(range(1, 3)),
+                                       pd.Float64Index(range(1, 3)),
+                                       pd.RangeIndex(1, 3)])
+    @pytest.mark.parametrize('scalar_td', [Timedelta(days=1),


these tests really belong with the scalar timedelta tests I think

Yah I thought about that and eventually decided that since the affected code was in the Index class, the test belonged in the Index tests. But either way works for me.

jreback · 2018-01-22T11:22:00Z

pandas/tests/indexes/test_numeric.py

@@ -26,6 +26,36 @@ def full_like(array, value):
    return ret


+class TestIndexArithmetic(object):


rename this class its index with timedelta scalar

add a comment. future readers are going to have no idea of where to find tests

…x_mul_tdi

jbrockmendel · 2018-01-22T19:04:02Z

Any idea what's special about the circleCI setup? I can't replicate the error there.

codecov · 2018-01-22T19:04:57Z

Codecov Report

Merging #19333 into master will decrease coverage by 0.02%.
The diff coverage is 95.23%.

@@            Coverage Diff             @@
##           master   #19333      +/-   ##
==========================================
- Coverage   91.61%   91.59%   -0.03%     
==========================================
  Files         150      150              
  Lines       48892    48911      +19     
==========================================
+ Hits        44792    44798       +6     
- Misses       4100     4113      +13

Flag	Coverage Δ
#multiple	`89.96% <95.23%> (-0.03%)`	⬇️
#single	`41.77% <14.28%> (-0.02%)`	⬇️

Impacted Files	Coverage Δ
pandas/core/indexes/range.py	`95.74% <100%> (+0.09%)`	⬆️
pandas/core/indexes/base.py	`96.41% <92.3%> (-0.04%)`	⬇️
pandas/plotting/_converter.py	`65.22% <0%> (-1.74%)`	⬇️

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update aa59954...9e3dec4. Read the comment docs.

…x_mul_tdi

jbrockmendel · 2018-01-24T05:04:56Z

@gfyoung any wisdom to offer about what makes circleci config special?

gfyoung · 2018-01-24T05:52:17Z

any wisdom to offer about what makes circleci config special?

@jbrockmendel : I wish 😉 I've generally found Circle and AppVeyor to be more PITA than Travis due to weird failures like these. AFAIK, Travis and Circle are quite similar, so unless you have some kind of flakiness in your tests, I'm not 100% sure why you are getting these results.

jbrockmendel · 2018-01-24T17:46:33Z

@gfyoung OK thanks. I'd rebase and re-push and hope that it sorts itself out, but don't want to put the strain on Travis.

gfyoung · 2018-01-24T17:50:09Z

but don't want to put the strain on Travis.

@jbrockmendel : No worries. I'm still kind of perplexed why Circle gets the completely wrong result when doing such a simple operation.

jreback · 2018-01-25T01:32:23Z

looks fine. ping when all green.

jreback · 2018-02-07T00:38:19Z

look on circleci on how to do this

jbrockmendel · 2018-02-10T01:55:41Z

https://circleci.com/docs/2.0/ssh-access-jobs/

Looks like I need to send you a public key and have you authorize it for me to debug.

jreback · 2018-02-10T16:51:44Z

you simply need to sign up on CircleCI, then you can run personal builds.

…x_mul_tdi

jbrockmendel · 2018-02-14T03:36:13Z

@gfyoung I made it onto CircleCI but couldn't replicate the test failure. Can I impose on you to give it a try? There's a beer and a high-five in it for you if that helps.

gfyoung · 2018-02-14T03:51:18Z

@jbrockmendel : Can you push those to me on GitHub? 😄 I restarted both AppVeyor and Circle for you. That being said, not sure yet what's causing the discrepancy. Might be something architecturally-related (e.g. int32 vs. int64).

jbrockmendel · 2018-02-14T04:03:12Z

Oh is it a 32 bit build thats broken? I hadn't noticed that.

jreback · 2018-02-15T12:28:05Z

pandas/core/indexes/base.py

+            values, other = other, values
+
+        with np.errstate(all='ignore'):
+            result = op(values, other)


can you add some comments (or expand the top ones), this code does a lot

jreback · 2018-02-15T12:29:50Z

small comments, is there an issue about this (or just PR)? rebase and ping on green.

jbrockmendel · 2018-02-15T15:52:01Z

small comments, is there an issue about this (or just PR)? rebase and ping on green.

Just the PR I think.

My attempt to SSH into CircleCI to debug the weird failing test failed. I'll try again at some point, hoping a fresh pair of eyeballs takes a crack at it in the interim.

…x_mul_tdi

jbrockmendel · 2018-02-21T02:50:23Z

I think I got this fixed, reopening.

jreback · 2018-02-21T03:00:59Z

what did it turn out to be?

jbrockmendel · 2018-02-21T03:06:27Z

What did it turn out to be?

I'll open an issue for the underlying cause because it might cause problems elsewhere: in sufficiently old numpy (the build that caught it used 1.9.2) we see the following:

value = np.timedelta64(86400000000000,'ns')
new_value = int(value)
assert new_value == value  # <-- in older numpy, this does not raise

As a result RangeIndex(np.timedelta64(1, 'ns'), np.timedelta64(10, 'ns'), np.timedelta64(1, 'ns')) doesn't raise and instead returns RangeIndex(1, 10, 1).

jreback · 2018-02-21T11:15:10Z

we do't need a new issue for this. see my comments there. rebase and I'll have a look.

jreback · 2018-02-21T11:16:05Z

value = np.timedelta64(86400000000000,'ns')
new_value = int(value)
assert new_value == value  # <-- in older numpy, this does not raise

there are many places in the code base where this is handled. This is the point of needing to check for timedeta before its checked for integer.

…x_mul_tdi

jbrockmendel · 2018-02-21T20:25:34Z

Rebase accomplished.

jreback · 2018-02-22T01:38:02Z

thanks! hey sometimes my stubborness pay off ..... thanks for sticking with it!

jbrockmendel added 8 commits January 20, 2018 22:29

update whatsnew

fecc906

add rmul test to timedeltas

7e8d5ff

Merge branch 'master' of https://github.com/pandas-dev/pandas into id…

d2423ca

…x_mul_tdi

handle Index * timedelta, with tests

1dd4ccf

whatsnew note

71811c8

division tests, fix typos

4c2c74b

Fix typo

d45ba92

restore whitespace

4abd61a

jreback requested changes Jan 21, 2018

View reviewed changes

jreback added Timedelta Timedelta data type Compat pandas objects compatability with Numpy or Python functions labels Jan 21, 2018

fill in GH issue

11424b0

Merge branch 'master' of https://github.com/pandas-dev/pandas into id…

ee48c8b

…x_mul_tdi

whitespace fixup from merge

1c97489

jreback reviewed Jan 21, 2018

View reviewed changes

arrange whatsnew notes

23b7f44

jreback requested changes Jan 22, 2018

View reviewed changes

Merge branch 'master' of https://github.com/pandas-dev/pandas into id…

086be08

…x_mul_tdi

comment, requested edits

6e04aba

Merge branch 'master' of https://github.com/pandas-dev/pandas into id…

1012939

…x_mul_tdi

jbrockmendel added 2 commits February 13, 2018 12:37

remove xfail

a46acb9

Merge branch 'master' of https://github.com/pandas-dev/pandas into id…

e6fa53b

…x_mul_tdi

jreback added this to the 0.23.0 milestone Feb 15, 2018

jreback requested changes Feb 15, 2018

View reviewed changes

jbrockmendel closed this Feb 16, 2018

jbrockmendel added 3 commits February 20, 2018 17:44

Merge branch 'master' of https://github.com/pandas-dev/pandas into id…

814e858

…x_mul_tdi

troubleshoot RangeIndex test failure

080c06c

fixup reversed op

22d155e

jbrockmendel reopened this Feb 21, 2018

jbrockmendel mentioned this pull request Feb 21, 2018

np.timedelta64 integer coercion in older numpy #19807

Closed

Merge branch 'master' of https://github.com/pandas-dev/pandas into id…

9e3dec4

…x_mul_tdi

jreback approved these changes Feb 22, 2018

View reviewed changes

jreback merged commit 3ab8623 into pandas-dev:master Feb 22, 2018

jbrockmendel deleted the idx_mul_tdi branch February 22, 2018 05:45

harisbal pushed a commit to harisbal/pandas that referenced this pull request Feb 28, 2018

Fix Index __mul__-like ops with timedelta scalars (pandas-dev#19333)

49bfc0b

		@@ -26,6 +26,36 @@ def full_like(array, value):
		return ret


		class TestIndexArithmetic(object):

Fix Index __mul__-like ops with timedelta scalars #19333

Fix Index __mul__-like ops with timedelta scalars #19333

Conversation

jbrockmendel commented Jan 21, 2018

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

jbrockmendel Jan 21, 2018 • edited Loading

Choose a reason for hiding this comment

jreback commented Jan 21, 2018

jbrockmendel commented Jan 21, 2018

jreback commented Jan 21, 2018

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

jbrockmendel commented Jan 22, 2018

codecov bot commented Jan 22, 2018 • edited Loading

Codecov Report

jbrockmendel commented Jan 24, 2018

gfyoung commented Jan 24, 2018

jbrockmendel commented Jan 24, 2018

gfyoung commented Jan 24, 2018 • edited Loading

jreback commented Jan 25, 2018

jreback commented Feb 7, 2018

jbrockmendel commented Feb 10, 2018

jreback commented Feb 10, 2018

jbrockmendel commented Feb 14, 2018

gfyoung commented Feb 14, 2018 • edited Loading

jbrockmendel commented Feb 14, 2018

Choose a reason for hiding this comment

jreback commented Feb 15, 2018

jbrockmendel commented Feb 15, 2018

jbrockmendel commented Feb 21, 2018

jreback commented Feb 21, 2018

jbrockmendel commented Feb 21, 2018

jreback commented Feb 21, 2018

jreback commented Feb 21, 2018

jbrockmendel commented Feb 21, 2018

jreback commented Feb 22, 2018

Fix Index mul-like ops with timedelta scalars #19333

Fix Index mul-like ops with timedelta scalars #19333

jbrockmendel Jan 21, 2018 •

edited

Loading

codecov bot commented Jan 22, 2018 •

edited

Loading

gfyoung commented Jan 24, 2018 •

edited

Loading

gfyoung commented Feb 14, 2018 •

edited

Loading