Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

BUG: Fix inconsistent ordering between left and right in merge #37406

Merged
merged 9 commits into from
Nov 21, 2020

Conversation

phofl
Copy link
Member

@phofl phofl commented Oct 25, 2020

Passed sort through for right join too. Did not see a reason why not.

On a related note: I think outer joins return a inconsistent order too.

df = pd.DataFrame({'a': [1, 0, 1]})

print(df.merge(df, on='a', how='outer', sort=False))
   a
0  1
1  1
2  1
3  1
4  0

instead of

   a
0  1
1  1
2  0
3  1
4  1

like left and right now. Would have to add a sort keyword to outer join. Should I open an issue for that?

@phofl phofl added the Reshaping Concat, Merge/Join, Stack/Unstack, Explode label Oct 25, 2020
pandas/core/reshape/merge.py Show resolved Hide resolved
@@ -2283,3 +2283,13 @@ def test_merge_join_categorical_multiindex():
expected = expected.drop(["Cat", "Int"], axis=1)
result = a.join(b, on=["Cat1", "Int1"])
tm.assert_frame_equal(expected, result)


@pytest.mark.parametrize("how", ["left", "right"])
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

sort=True is ok? (do we test this)? can you locate this test function near similar tests.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Sort True was the previous default behavior. Added tests to show behavior. Moved the tests up a bit right below a nosort test

@jreback
Copy link
Contributor

jreback commented Nov 4, 2020

can you merge master and will look

� Conflicts:
�	doc/source/whatsnew/v1.2.0.rst
@phofl
Copy link
Member Author

phofl commented Nov 4, 2020

merged master

Copy link
Contributor

@jreback jreback left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

could do this for 1.2 if you can update and merge master

pandas/core/reshape/merge.py Show resolved Hide resolved
@phofl
Copy link
Member Author

phofl commented Nov 19, 2020

Merged master too

@jreback jreback added this to the 1.2 milestone Nov 20, 2020
Copy link
Contributor

@jreback jreback left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

small comment ping on green

@@ -712,6 +712,7 @@ Reshaping
- Fixed regression in :func:`merge` on merging DatetimeIndex with empty DataFrame (:issue:`36895`)
- Bug in :meth:`DataFrame.apply` not setting index of return value when ``func`` return type is ``dict`` (:issue:`37544`)
- Bug in :func:`concat` resulting in a ``ValueError`` when at least one of both inputs had a non-unique index (:issue:`36263`)
- Bug in :meth:`df.merge() <pandas.DataFrame.merge>` returning inconsistent ordering in result for ``how=right`` and ``how=left`` (:issue:`35382`)
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

you don't need to add the <...> instead do this like `:meth:`DataFrame.merge`

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

also add :meth:`pandas.merge`

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Done

@jreback jreback merged commit 3d5e65d into pandas-dev:master Nov 21, 2020
@jreback
Copy link
Contributor

jreback commented Nov 21, 2020

thanks @phofl very nice!

@jreback
Copy link
Contributor

jreback commented Nov 21, 2020

this migtht close #31951, #22449, #17257

if any of these, would take a test and we can close

@phofl phofl deleted the 35382 branch November 21, 2020 23:43
@phofl
Copy link
Member Author

phofl commented Nov 21, 2020

Thx

No they are not related. but #22449 can be closed probably, after we improved the Error Reporting here this should definitely fail as said by @TomAugspurger

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Reshaping Concat, Merge/Join, Stack/Unstack, Explode
Projects
None yet
Development

Successfully merging this pull request may close these issues.

BUG: Inconsistent ordering of rows when merging with how=left and how=right
2 participants