ENH: groupby missing data in index #28097

proost · 2019-08-22T17:34:29Z

closes Groupby on multiindex with missing data in group keys raises IndexError #20519
tests added / passed
passes black pandas
passes git diff upstream/master -u -- "*.py" | flake8 --diff
whatsnew entry

.groupby intialize "grouper" by _get_grouper_for_level. Base on "code", "level_index" fills with non-NA value if there is NA value. Problem is if there are only NA values, "level_index" can't fill with values.
So, check "level_index" whether all "level_index" values is NA. then "level_index" are all NA values, fill with NaN

proost · 2019-08-23T03:46:46Z

@jreback could you merge this pr?

jreback

@proost this needs review and will file comments at some point

WillAyd

Haven't deeply reviewed change but I see a comment a few lines up that says "Handle NA" - is that comment or block of code not applicable? Would like to refactor instead of adding logic if possible

pandas/tests/indexes/test_base.py

doc/source/whatsnew/v0.25.1.rst

jreback

pls merge master and address comments

pandas/core/indexes/multi.py

pandas/tests/indexes/test_base.py

pep8speaks · 2019-09-10T05:44:50Z

Hello @proost! Thanks for updating this PR. We checked the lines you've touched for PEP 8 issues, and found:

There are currently no PEP 8 issues detected in this Pull Request. Cheers! 🍻

Comment last updated at 2019-10-25 04:59:47 UTC

doc/source/whatsnew/v1.0.0.rst

pandas/tests/groupby/test_grouping.py

WillAyd

Looking pretty good

doc/source/whatsnew/v1.0.0.rst

pandas/tests/groupby/test_grouping.py

jbrockmendel · 2019-10-11T22:59:49Z

pandas/tests/groupby/test_grouping.py

+    def test_groupby_level_index_value_all_na(self):
+        # issue 20519
+        df = pd.DataFrame(
+            [["x", np.nan, 10], [None, np.nan, 20]], columns=["A", "B", "C"]


If I cast df["B"] = df["B"].astype("datetime64") before doing the set_index, I get a different error on the groupby call (in master). Should that be fixed by this PR? If so, please test.

side-note, I'd find this easier to follow in smaller steps:

df = pd.DataFrame(...) df = df.set_index(["A", "B"]) gb = df.groupby(level=["A", "B"]) result = gb.sum()

Especially relevant as it is the gb = ... line that raises in master

@jbrockmendel
I'm bit confused. you df["B"] = df["B"].astype("datetime64") means df["B"] = df["B"].astype("datetime64[ns]") write?
in master,
df = pd.DataFrame(...)
df["B"] = df["B"].astype("datetime64[ns]")
df = df.set_index(["A", "B"])
gb = df.groupby(level=["A", "B"])
and
df = pd.DataFrame(...)
df = df.set_index(["A", "B"])
gb = df.groupby(level=["A", "B"])
raise same IndexError :cannot do a non-empty take from an empty axes.

I'm bit confused. you df["B"] = df["B"].astype("datetime64") means df["B"] = df["B"].astype("datetime64[ns]") write?

Yes, I meant to write "datetime64[ns]" and not just "datetime64".

raise same IndexError

Huh, not sure how I got to a different error. My bad.

WillAyd · 2019-10-22T01:36:28Z

@proost can you fix up merge conflict?

…exError (pandas-dev#20519) * if all the values in a level of a MultiIndex were missing, fill with numpy nan

jreback · 2019-10-25T12:54:13Z

pandas/core/indexes/multi.py

+        if len(level_index):
+            grouper = level_index.take(codes)
+        else:
+            grouper = level_index.take(codes, fill_value=True)


hmm, can you eliminate the branch, and just always pass fill_value=True ?

If remove the branch, for example,

df = DataFrame([["x", 1, 10], ["y", 2, 20]], columns=["A", "B", "C"]).set_index(["A", "B"]) result = df.groupby(level=["A", "B"]).sum()

raise exception.

actually i think this is incorrect

@WillAyd wasn’t done here

this branch can be simplified i think

Sorry misread the approval above

@proost do you mind simplifying this in a follow up PR?

WillAyd · 2019-10-25T16:35:00Z

Thanks @proost

proost · 2019-10-25T16:41:53Z

@jreback @WillAyd @jbrockmendel @TomAugspurger
Thank you for you review and feedback

…ndexing-1row-df * upstream/master: (194 commits) DOC Remove Python 2 specific comments from documentation (pandas-dev#31198) Follow up PR: pandas-dev#28097 Simplify branch statement (pandas-dev#29243) BUG: DatetimeIndex.snap incorrectly setting freq (pandas-dev#31188) Move DataFrame.info() to live with similar functions (pandas-dev#31317) ENH: accept a dictionary in plot colors (pandas-dev#31071) PERF: add shortcut to Timestamp constructor (pandas-dev#30676) CLN/MAINT: Clean and annotate stata reader and writers (pandas-dev#31072) REF: define _get_slice_axis in correct classes (pandas-dev#31304) BUG: DataFrame.floordiv(ser, axis=0) not matching column-wise bheavior (pandas-dev#31271) PERF: optimize is_scalar, is_iterator (pandas-dev#31294) BUG: Series rolling count ignores min_periods (pandas-dev#30923) xfail sparse warning; closes pandas-dev#31310 (pandas-dev#31311) REF: DatetimeIndex.get_value wrap DTI.get_loc (pandas-dev#31314) CLN: internals.managers (pandas-dev#31316) PERF: avoid copies if possible in fill_binop (pandas-dev#31300) Add test for multiindex json (pandas-dev#31307) BUG: passing TDA and wrong freq to TimedeltaIndex (pandas-dev#31268) BUG: inconsistency between PeriodIndex.get_value vs get_loc (pandas-dev#31172) CLN: remove _set_subtyp (pandas-dev#31301) CI: Updated version of macos image (pandas-dev#31292) ...

… branch statement

…31689) Co-authored-by: proost <[email protected]>

jreback requested changes Aug 23, 2019

View reviewed changes

simonjayhawkins added Groupby Missing-data np.nan, pd.NaT, pd.NA, dropna, isnull, interpolate MultiIndex labels Aug 25, 2019

WillAyd requested changes Aug 26, 2019

View reviewed changes

pandas/tests/indexes/test_base.py Outdated Show resolved Hide resolved

pandas/tests/indexes/test_base.py Outdated Show resolved Hide resolved

WillAyd added this to the 1.0 milestone Aug 26, 2019

WillAyd requested changes Aug 26, 2019

View reviewed changes

doc/source/whatsnew/v0.25.1.rst Outdated Show resolved Hide resolved

jreback requested changes Sep 8, 2019

View reviewed changes

pandas/core/indexes/multi.py Outdated Show resolved Hide resolved

pandas/tests/indexes/test_base.py Outdated Show resolved Hide resolved

pandas/tests/indexes/test_base.py Outdated Show resolved Hide resolved

jreback removed this from the 1.0 milestone Sep 8, 2019

proost force-pushed the update-groupby branch from 85b9eef to 6576242 Compare September 10, 2019 05:44

proost force-pushed the update-groupby branch 2 times, most recently from 4a7e25e to ab639ca Compare September 10, 2019 06:58

TomAugspurger reviewed Sep 10, 2019

View reviewed changes

doc/source/whatsnew/v1.0.0.rst Outdated Show resolved Hide resolved

pandas/tests/groupby/test_grouping.py Outdated Show resolved Hide resolved

pandas/tests/groupby/test_grouping.py Outdated Show resolved Hide resolved

proost force-pushed the update-groupby branch 8 times, most recently from dd5ea8b to 892126f Compare September 14, 2019 15:44

proost requested review from jreback and WillAyd September 16, 2019 01:32

WillAyd reviewed Sep 16, 2019

View reviewed changes

pandas/tests/groupby/test_grouping.py Show resolved Hide resolved

WillAyd requested changes Sep 17, 2019

View reviewed changes

doc/source/whatsnew/v1.0.0.rst Outdated Show resolved Hide resolved

pandas/tests/groupby/test_grouping.py Outdated Show resolved Hide resolved

proost force-pushed the update-groupby branch from 5b89726 to 52cf65a Compare September 18, 2019 06:33

proost requested a review from WillAyd September 18, 2019 08:09

WillAyd added this to the 1.0 milestone Sep 18, 2019

jbrockmendel reviewed Oct 11, 2019

View reviewed changes

proost force-pushed the update-groupby branch from 8d26f5a to 85e066b Compare October 22, 2019 03:19

EHN: Groupby on multiindex with missing data in group keys raises Ind…

6a01c73

…exError (pandas-dev#20519) * if all the values in a level of a MultiIndex were missing, fill with numpy nan

proost force-pushed the update-groupby branch 2 times, most recently from 6b9aef6 to 6a01c73 Compare October 25, 2019 01:57

proost added 2 commits October 25, 2019 10:59

Merge branch 'master' into update-groupby

454de6b

Update v1.0.0.rst

ef08517

jreback approved these changes Oct 25, 2019

View reviewed changes

jreback reviewed Oct 25, 2019

View reviewed changes

WillAyd merged commit f03ed62 into pandas-dev:master Oct 25, 2019

proost deleted the update-groupby branch October 25, 2019 16:40

proost added a commit to proost/pandas that referenced this pull request Oct 27, 2019

Follow up PR: pandas-dev#28097 Simplify branch statement

c6e8611

proost mentioned this pull request Oct 27, 2019

Follow up PR: #28097 Simplify branch statement #29243

Merged

proost added a commit to proost/pandas that referenced this pull request Oct 28, 2019

Follow up PR: pandas-dev#28097 Simplify branch statement

e97d879

proost added a commit to proost/pandas that referenced this pull request Oct 28, 2019

Follow up PR: pandas-dev#28097 Simplify branch statement

a212065

HawkinsBA pushed a commit to HawkinsBA/pandas that referenced this pull request Oct 29, 2019

ENH: groupby missing data in index (pandas-dev#28097)

946781d

Reksbril pushed a commit to Reksbril/pandas that referenced this pull request Nov 18, 2019

ENH: groupby missing data in index (pandas-dev#28097)

eaeb9f3

proost added a commit to proost/pandas that referenced this pull request Dec 19, 2019

ENH: groupby missing data in index (pandas-dev#28097)

1688fac

proost added a commit to proost/pandas that referenced this pull request Dec 19, 2019

ENH: groupby missing data in index (pandas-dev#28097)

be317dd

bongolegend pushed a commit to bongolegend/pandas that referenced this pull request Jan 1, 2020

ENH: groupby missing data in index (pandas-dev#28097)

0755533

proost added a commit to proost/pandas that referenced this pull request Jan 2, 2020

Follow up PR: pandas-dev#28097 Simplify branch statement

bc71f05

proost added a commit to proost/pandas that referenced this pull request Jan 15, 2020

Follow up PR: pandas-dev#28097 Simplify branch statement

29977dc

jreback pushed a commit that referenced this pull request Jan 26, 2020

Follow up PR: #28097 Simplify branch statement (#29243)

8e1dd1e

meeseeksmachine mentioned this pull request Feb 5, 2020

Backport PR #29243 on branch 1.0.x (Follow up PR: #28097 Simplify branch statement) #31689

Merged

meeseeksmachine pushed a commit to meeseeksmachine/pandas that referenced this pull request Feb 5, 2020

Backport PR pandas-dev#29243: Follow up PR: pandas-dev#28097 Simplify…

55ff6c5

… branch statement

jorisvandenbossche pushed a commit that referenced this pull request Feb 5, 2020

Backport PR #29243: Follow up PR: #28097 Simplify branch statement (#…

3e8a4c2

…31689) Co-authored-by: proost <[email protected]>

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

ENH: groupby missing data in index #28097

ENH: groupby missing data in index #28097

proost commented Aug 22, 2019 •

edited

Loading

proost commented Aug 23, 2019

jreback left a comment

WillAyd left a comment

jreback left a comment

pep8speaks commented Sep 10, 2019 •

edited

Loading

WillAyd left a comment

jbrockmendel Oct 11, 2019

proost Oct 13, 2019 •

edited

Loading

jbrockmendel Oct 16, 2019

WillAyd commented Oct 22, 2019

jreback Oct 25, 2019

proost Oct 25, 2019

jreback Oct 25, 2019

WillAyd Oct 25, 2019

WillAyd Oct 25, 2019

WillAyd commented Oct 25, 2019

proost commented Oct 25, 2019

ENH: groupby missing data in index #28097

ENH: groupby missing data in index #28097

Conversation

proost commented Aug 22, 2019 • edited Loading

proost commented Aug 23, 2019

jreback left a comment

Choose a reason for hiding this comment

WillAyd left a comment

Choose a reason for hiding this comment

jreback left a comment

Choose a reason for hiding this comment

pep8speaks commented Sep 10, 2019 • edited Loading

Comment last updated at 2019-10-25 04:59:47 UTC

WillAyd left a comment

Choose a reason for hiding this comment

jbrockmendel Oct 11, 2019

Choose a reason for hiding this comment

proost Oct 13, 2019 • edited Loading

Choose a reason for hiding this comment

jbrockmendel Oct 16, 2019

Choose a reason for hiding this comment

WillAyd commented Oct 22, 2019

jreback Oct 25, 2019

Choose a reason for hiding this comment

proost Oct 25, 2019

Choose a reason for hiding this comment

jreback Oct 25, 2019

Choose a reason for hiding this comment

WillAyd Oct 25, 2019

Choose a reason for hiding this comment

WillAyd Oct 25, 2019

Choose a reason for hiding this comment

WillAyd commented Oct 25, 2019

proost commented Oct 25, 2019

proost commented Aug 22, 2019 •

edited

Loading

pep8speaks commented Sep 10, 2019 •

edited

Loading

proost Oct 13, 2019 •

edited

Loading