BUG: Series.any() and .all() don't return bool values if dtype=object #30416

ShaharNaveh · 2019-12-22T23:02:28Z

closes Series.any() and .all() don't return bool values if dtype=object #12863
tests added / passed
passes black pandas
passes git diff upstream/master -u -- "*.py" | flake8 --diff
whatsnew entry

pandas/core/nanops.py

pep8speaks · 2019-12-23T12:47:40Z

Hello @MomIsBestFriend! Thanks for updating this PR. We checked the lines you've touched for PEP 8 issues, and found:

There are currently no PEP 8 issues detected in this Pull Request. Cheers! 🍻

Comment last updated at 2020-01-12 16:37:00 UTC

jreback

lgtm. can you add a whatsnew note. i think this deserves a small subsection (even though its a bug fix), you can put in api-breaking changes, just show the before and after (model after some existing changes like this)

pandas/core/nanops.py

ShaharNaveh · 2019-12-24T22:08:25Z

The only implementation I can think of right now, is to create a function to iterate over the values and returns a bool if it the any/all the values elevated to to True.

tl;dr
Implement any and all methods.

jbrockmendel · 2019-12-25T00:02:08Z

pandas/core/nanops.py

+    # GH #12863
+    # Checking if the `axis` is None because numpy
+    # doesn't handle ``any`` and ``all`` on
+    # object arrays correclty. see


correclty -> correctly. small preference for this to go inside the "if", not sure how widely shared that is

jbrockmendel · 2019-12-25T00:02:46Z

pandas/tests/reductions/test_reductions.py

        assert s1.all(skipna=True)
-        assert np.isnan(s2.any(skipna=False))  # nan || False => nan
+        assert s1.any(skipna=True)


pls add the GH ref here too

jbrockmendel · 2019-12-25T00:03:35Z

pandas/tests/reductions/test_reductions.py

-        # Alternative types, with implicit 'object' dtype.
-        s = Series(["abc", True])
-        assert "abc" == s.any()  # 'abc' || True => 'abc'
+        # GH 12863


might as well make this a separate test test_all_any_object_dtype

jbrockmendel · 2019-12-25T00:04:44Z

Does this bugfix affect DataFrame at all? if so, test for that?

optional: could test the affected nanops functions directly in test_nanops.

jbrockmendel · 2019-12-25T00:26:11Z

what case is breaking that necessitates the brute force patch?

ShaharNaveh · 2019-12-25T00:50:18Z

what case is breaking that necessitates the brute force patch?

The original test case (from #12863)

pd.Series(index=range(5), data=['a', 'b', 'c', 'd', 'e'])

ShaharNaveh · 2019-12-25T23:34:15Z

I can't figure it out, when the original case is working, every test is failing.
I can xfail them all, but it's just seems wrong, that so many fails.

jreback · 2019-12-26T00:37:53Z

pandas/core/nanops.py

+    if is_object_dtype(dtype) and axis is None:
+        output = values.any()
+    else:
+        output = values.any(axis)


you can just return directly from here.

jreback · 2019-12-26T00:38:01Z

pandas/core/nanops.py

+    else:
+        output = values.any(axis)
+
+    if isinstance(output, bool):


when is this NOT true?

That's exactly the bug, for example (from the original issue).

master:

>>> import pandas as pd >>> s = pd.Series(index=range(5), data=['a', 'b', 'c', 'd', 'e'], dtype=object) >>> s.any() 'a' >>> s.all() 'e'

This branch (with brute force function):

>>> import pandas as pd >>> s = pd.Series(index=range(5), data=['a', 'b', 'c', 'd', 'e'], dtype=object) >>> s.any() True >>> s.all() True

jreback · 2020-01-01T17:25:48Z

can you merge master and we'll take a look

ShaharNaveh · 2020-01-01T18:50:15Z

can you merge master and we'll take a look

I have merged master @jreback

jreback · 2020-01-01T20:49:05Z

@MomIsBestFriend so the tests are failing, ping when green.

ShaharNaveh · 2020-01-01T21:17:55Z

@MomIsBestFriend so the tests are failing, ping when green.

@jreback I'm really not sure what to do, I see pretty much everywhere the error message of:

AttributeError: type object 'bool' has no attribute 'any'

No idea on where to even start. should I change the nanany/nanall functions?

or change about 30 - 50 tests, it's just seems wrong to do so.

jreback · 2020-01-01T21:19:04Z

@MomIsBestFriend so the tests are failing, ping when green.

@jreback I'm really not sure what to do, I see pretty much everywhere the error message of:
AttributeError: type object 'bool' has no attribute 'any'
No idea on where to even start. should I change the nanany/nanall functions?

or change about 30 - 50 tests, it's just seems wrong to do so.

try using np.any

ShaharNaveh · 2020-01-01T21:23:39Z

@MomIsBestFriend so the tests are failing, ping when green.

@jreback I'm really not sure what to do, I see pretty much everywhere the error message of:
AttributeError: type object 'bool' has no attribute 'any'
No idea on where to even start. should I change the nanany/nanall functions?
or change about 30 - 50 tests, it's just seems wrong to do so.
try using np.any

This also doesn't work.

jbrockmendel · 2020-01-01T23:01:09Z

This also doesn't work.

@MomIsBestFriend now that #29847 has been merged, I'm planning to do a significant cleanup of DataFrame._reduce and core.nanops. So I think the path forward is for you and I to trade tasks. Up for something new?

ShaharNaveh · 2020-01-02T00:31:16Z

Up for something new?

Yes just take it away from me 😆

ShaharNaveh · 2020-01-12T16:52:20Z

Closing this, because this I don't know how to fix it without breaking everything else.

gimseng · 2020-10-08T13:32:15Z

Did the issue eventually got fixed? Can someone tldr the thread? Thanks !

jbrockmendel reviewed Dec 23, 2019

View reviewed changes

pandas/core/nanops.py Outdated Show resolved Hide resolved

alimcmaster1 added Dtype Conversions Unexpected or buggy dtype conversions Compat pandas objects compatability with Numpy or Python functions labels Dec 23, 2019

ShaharNaveh force-pushed the COMPAT-fix-12863 branch 2 times, most recently from f9037cf to 876b763 Compare December 23, 2019 12:47

ShaharNaveh force-pushed the COMPAT-fix-12863 branch 3 times, most recently from 78da081 to 48f08c0 Compare December 23, 2019 13:14

ShaharNaveh marked this pull request as ready for review December 23, 2019 13:47

jreback changed the title ~~BUG: Fix for #12863~~ Series.any() and .all() don't return bool values if dtype=object Dec 23, 2019

jreback changed the title ~~Series.any() and .all() don't return bool values if dtype=object~~ BUG: Series.any() and .all() don't return bool values if dtype=object Dec 23, 2019

jreback requested changes Dec 23, 2019

View reviewed changes

pandas/core/nanops.py Show resolved Hide resolved

jreback added this to the 1.0 milestone Dec 23, 2019

ShaharNaveh force-pushed the COMPAT-fix-12863 branch 2 times, most recently from 35391e9 to 650c906 Compare December 24, 2019 21:47

ShaharNaveh force-pushed the COMPAT-fix-12863 branch from 650c906 to 00c9b50 Compare December 24, 2019 23:56

jbrockmendel reviewed Dec 25, 2019

View reviewed changes

jbrockmendel mentioned this pull request Dec 25, 2019

PERF: perform reductions block-wise #29847

Merged

ShaharNaveh force-pushed the COMPAT-fix-12863 branch from e87ba0a to f774031 Compare December 25, 2019 00:44

ShaharNaveh force-pushed the COMPAT-fix-12863 branch 2 times, most recently from d6996ca to cf72a99 Compare December 25, 2019 22:21

jreback requested changes Dec 26, 2019

View reviewed changes

jreback removed this from the 1.0 milestone Jan 3, 2020

ShaharNaveh force-pushed the COMPAT-fix-12863 branch 2 times, most recently from 7864171 to 2217be0 Compare January 12, 2020 16:34

BUG: fix the return value of .all() and .any() on pd.Series

3492806

ShaharNaveh force-pushed the COMPAT-fix-12863 branch from 2217be0 to 3492806 Compare January 12, 2020 16:36

ShaharNaveh closed this Jan 12, 2020

ShaharNaveh deleted the COMPAT-fix-12863 branch January 12, 2020 16:52

gimseng mentioned this pull request Oct 8, 2020

BUG: any() and all() behavior on string series is different from python #36880

Closed

3 tasks

gimseng mentioned this pull request Oct 8, 2020

BUG: dataframe.any() method behaves differently for empty rows and columns #35450

Closed

3 tasks

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

BUG: Series.any() and .all() don't return bool values if dtype=object #30416

BUG: Series.any() and .all() don't return bool values if dtype=object #30416

ShaharNaveh commented Dec 22, 2019 •

edited

Loading

pep8speaks commented Dec 23, 2019 •

edited

Loading

jreback left a comment

ShaharNaveh commented Dec 24, 2019 •

edited

Loading

jbrockmendel Dec 25, 2019

jbrockmendel Dec 25, 2019

jbrockmendel Dec 25, 2019

jbrockmendel commented Dec 25, 2019

jbrockmendel commented Dec 25, 2019

ShaharNaveh commented Dec 25, 2019

ShaharNaveh commented Dec 25, 2019

jreback Dec 26, 2019

jreback Dec 26, 2019

ShaharNaveh Dec 28, 2019

jreback commented Jan 1, 2020

ShaharNaveh commented Jan 1, 2020

jreback commented Jan 1, 2020

ShaharNaveh commented Jan 1, 2020 •

edited

Loading

jreback commented Jan 1, 2020

ShaharNaveh commented Jan 1, 2020

jbrockmendel commented Jan 1, 2020

ShaharNaveh commented Jan 2, 2020

ShaharNaveh commented Jan 12, 2020

gimseng commented Oct 8, 2020

BUG: Series.any() and .all() don't return bool values if dtype=object #30416

BUG: Series.any() and .all() don't return bool values if dtype=object #30416

Conversation

ShaharNaveh commented Dec 22, 2019 • edited Loading

pep8speaks commented Dec 23, 2019 • edited Loading

Comment last updated at 2020-01-12 16:37:00 UTC

jreback left a comment

Choose a reason for hiding this comment

ShaharNaveh commented Dec 24, 2019 • edited Loading

jbrockmendel Dec 25, 2019

Choose a reason for hiding this comment

jbrockmendel Dec 25, 2019

Choose a reason for hiding this comment

jbrockmendel Dec 25, 2019

Choose a reason for hiding this comment

jbrockmendel commented Dec 25, 2019

jbrockmendel commented Dec 25, 2019

ShaharNaveh commented Dec 25, 2019

ShaharNaveh commented Dec 25, 2019

jreback Dec 26, 2019

Choose a reason for hiding this comment

jreback Dec 26, 2019

Choose a reason for hiding this comment

ShaharNaveh Dec 28, 2019

Choose a reason for hiding this comment

jreback commented Jan 1, 2020

ShaharNaveh commented Jan 1, 2020

jreback commented Jan 1, 2020

ShaharNaveh commented Jan 1, 2020 • edited Loading

jreback commented Jan 1, 2020

ShaharNaveh commented Jan 1, 2020

jbrockmendel commented Jan 1, 2020

ShaharNaveh commented Jan 2, 2020

ShaharNaveh commented Jan 12, 2020

gimseng commented Oct 8, 2020

ShaharNaveh commented Dec 22, 2019 •

edited

Loading

pep8speaks commented Dec 23, 2019 •

edited

Loading

ShaharNaveh commented Dec 24, 2019 •

edited

Loading

ShaharNaveh commented Jan 1, 2020 •

edited

Loading