Frame ops prelims - de-duplicate, remove unused kwargs #19522

jbrockmendel · 2018-02-03T03:26:21Z

Starting in on making DataFrame ops consistent with Series and Index ops. As a first pass, this goes through and removes some duplicate code and unused arguments.

Changes logic in exactly two corner cases: arithmetic operations frame.op(series, axis=None, fill_value=not_none) where one of series or frame has length zero. axis needs to be specifically passed as None to hit the changed case.

ser = pd.Series([])
df = pd.DataFrame([[1, 2], [3, 4]])

>>> df.add(ser, axis=None, fill_value='E')
>>> ser.to_frame().sub(df[0], axis=None, fill_value=3)

will now raise instead of returning

>>> df.add(ser, axis=None, fill_value='E')
    0   1
0 NaN NaN
1 NaN NaN

>>> ser.to_frame().sub(df[0], axis=None, fill_value=3)
Empty DataFrame
Columns: [0]
Index: []

codecov · 2018-02-03T04:19:41Z

Codecov Report

Merging #19522 into master will decrease coverage by <.01%.
The diff coverage is 90.9%.

@@            Coverage Diff             @@
##           master   #19522      +/-   ##
==========================================
- Coverage    91.6%    91.6%   -0.01%     
==========================================
  Files         150      150              
  Lines       48750    48736      -14     
==========================================
- Hits        44657    44644      -13     
+ Misses       4093     4092       -1

Flag	Coverage Δ
#multiple	`89.97% <90.9%> (-0.01%)`	⬇️
#single	`41.74% <50%> (-0.01%)`	⬇️

Impacted Files	Coverage Δ
pandas/core/sparse/frame.py	`94.83% <100%> (-0.02%)`	⬇️
pandas/core/indexes/base.py	`96.43% <100%> (-0.02%)`	⬇️
pandas/core/ops.py	`95.22% <100%> (ø)`	⬆️
pandas/core/frame.py	`97.42% <80%> (-0.01%)`	⬇️
pandas/core/reshape/merge.py	`94.22% <0%> (ø)`	⬆️
pandas/io/parsers.py	`95.56% <0%> (+0.06%)`	⬆️

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update ed10bf6...85d501c. Read the comment docs.

jreback · 2018-02-04T16:21:11Z

pandas/core/frame.py

-                                          fill_value=fill_value,
-                                          try_cast=try_cast)
+        else:
+            if len(other) == 0:


jreback · 2018-02-04T16:21:16Z

pandas/core/frame.py

-                              fill_value=None, try_cast=True):
-        if len(other) == 0:
-            return self * np.nan
+            if len(self) == 0:


jreback · 2018-02-04T16:21:30Z

pandas/core/frame.py

-        if len(other) == 0:
-            return self * np.nan
+            if len(self) == 0:
+                # Ambiguous case, use _series so works with DataFrame


what is this comment?

Not totally sure. Possibly a holdover from higher-dim implementation?

Blame shows that this comment was introduced in c35a39d in 2011. Looks like it replaced a comment # Ambiguous case, use _series so works with DataMatrix. @wesm is this still relevant? Suggestions for fleshing this out?

jreback · 2018-02-04T16:22:02Z

pandas/core/frame.py

@@ -3960,52 +3959,40 @@ def f(i):

    def _combine_series(self, other, func, fill_value=None, axis=None,
                        level=None, try_cast=True):
+        if fill_value is not None:
+            raise NotImplementedError("fill_value %r not supported." %


is this hit in tests?

can you use .format() here instead

Yes to both. tests.frame.test_operators.TestDataFrameOperators.test_arith_flex_frame:

with tm.assert_raises_regex(NotImplementedError, 'fill_value'): self.frame.add(self.frame.iloc[0], fill_value=3)

…s-frame3

jreback · 2018-02-06T01:10:19Z

so you need some tests to validate your claims at the top of the PR

Changes logic in exactly two corner cases: arithmetic operations frame.op(series, axis=None,

and a release note.

…s-frame3

jreback · 2018-02-06T11:21:30Z

doc/source/whatsnew/v0.23.0.txt

@@ -581,6 +581,7 @@ Numeric
 - Bug in :class:`Index` multiplication and division methods where operating with a ``Series`` would return an ``Index`` object instead of a ``Series`` object (:issue:`19042`)
 - Bug in the :class:`DataFrame` constructor in which data containing very large positive or very large negative numbers was causing ``OverflowError`` (:issue:`18584`)
 - Bug in :class:`Index` constructor with ``dtype='uint64'`` where int-like floats were not coerced to :class:`UInt64Index` (:issue:`18400`)
+- Bug in :class:`DataFrame` flex arithmetic methods that failed to raise ``NotImplementedError` in cases where either argument has length zero (:issue:`19522`)


need double backticks on NotImplementedError. can you be slightly more descriptive here.

a user is reading this and would want to know what if anything they need to change.

jreback · 2018-02-06T11:22:14Z

pandas/core/indexes/base.py

@@ -80,6 +80,26 @@ def _try_get_item(x):
        return x


+def _make_invalid_op(name):


consider moving this function to core/ops (and parameterizing so you can use it for index/series)

jreback · 2018-02-06T11:22:55Z

pandas/tests/frame/test_operators.py

@@ -451,6 +451,18 @@ def test_arith_flex_frame(self):
        with tm.assert_raises_regex(NotImplementedError, 'fill_value'):
            self.frame.add(self.frame.iloc[0], axis='index', fill_value=3)

+    def test_arith_flex_zero_len_raises(self):


shouldn't these be in test_arithmetic?

Eventually yes. The more important grouping for now is that this belongs with the test above it, which I plan to split/parametrize before moving to test_arithmetic

jreback · 2018-02-06T11:23:50Z

pandas/tests/frame/test_operators.py

+            df.add(ser, fill_value='E')
+
+        with tm.assert_raises_regex(NotImplementedError, 'fill_value'):
+            ser.to_frame().sub(df[0], axis=None, fill_value=3)


just create the required frame here rather than using ser.to_frame()

jreback · 2018-02-06T11:24:09Z

pandas/tests/frame/test_operators.py

+        df = pd.DataFrame([[1, 2], [3, 4]])
+
+        with tm.assert_raises_regex(NotImplementedError, 'fill_value'):
+            df.add(ser, fill_value='E')


this should raise ValueError, not NIE.

The existing implementation raises NIE outside of these corner cases. For the time being I'm assuming there's a reason for that.

jreback · 2018-02-07T12:32:40Z

thanks

)

jbrockmendel added 2 commits February 2, 2018 19:03

de-duplicate, remove unused arguments

a8cc5c0

avoid nameerror

ce3e616

gfyoung added Internals Related to non-user accessible pandas implementation Clean labels Feb 3, 2018

jreback requested changes Feb 4, 2018

View reviewed changes

jbrockmendel added 2 commits February 4, 2018 09:05

Merge branch 'master' of https://github.com/pandas-dev/pandas into op…

cffcf31

…s-frame3

requested edits

6c0811a

jbrockmendel added 2 commits February 5, 2018 18:32

test, whatsnew note

2de0268

Merge branch 'master' of https://github.com/pandas-dev/pandas into op…

92749dc

…s-frame3

jreback requested changes Feb 6, 2018

View reviewed changes

jbrockmendel added 2 commits February 6, 2018 08:14

flesh out whatsnew

98fdcef

Fixup columns changed

85d501c

jreback added this to the 0.23.0 milestone Feb 7, 2018

jreback approved these changes Feb 7, 2018

View reviewed changes

jreback merged commit a7d1103 into pandas-dev:master Feb 7, 2018

jbrockmendel deleted the ops-frame3 branch February 11, 2018 21:38

harisbal pushed a commit to harisbal/pandas that referenced this pull request Feb 28, 2018

Frame ops prelims - de-duplicate, remove unused kwargs (pandas-dev#19522

7a19781

)

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Frame ops prelims - de-duplicate, remove unused kwargs #19522

Frame ops prelims - de-duplicate, remove unused kwargs #19522

jbrockmendel commented Feb 3, 2018

codecov bot commented Feb 3, 2018 •

edited

Loading

jreback Feb 4, 2018

jreback Feb 4, 2018

jreback Feb 4, 2018

jbrockmendel Feb 4, 2018

jbrockmendel Feb 5, 2018

jreback Feb 4, 2018

jreback Feb 4, 2018

jbrockmendel Feb 4, 2018

jreback commented Feb 6, 2018

jreback Feb 6, 2018

jreback Feb 6, 2018

jreback Feb 6, 2018

jreback Feb 6, 2018

jbrockmendel Feb 6, 2018

jreback Feb 6, 2018

jbrockmendel Feb 6, 2018

jreback Feb 6, 2018

jbrockmendel Feb 6, 2018

jreback commented Feb 7, 2018

		@@ -80,6 +80,26 @@ def _try_get_item(x):
		return x


		def _make_invalid_op(name):

Frame ops prelims - de-duplicate, remove unused kwargs #19522

Frame ops prelims - de-duplicate, remove unused kwargs #19522

Conversation

jbrockmendel commented Feb 3, 2018

codecov bot commented Feb 3, 2018 • edited Loading

Codecov Report

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

jreback commented Feb 6, 2018

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

jreback commented Feb 7, 2018

codecov bot commented Feb 3, 2018 •

edited

Loading