ENH: Add duplicated/drop_duplicates to Index

[sinhrks]

Closes #4060.

idx = pd.Index([1, 2, 3, 4, 1, 2])
idx.duplicated()
# Index([False, False, False, False, True, True], dtype='bool')
idx.drop_duplicates()
# Int64Index([1, 2, 3, 4], dtype='int64')

idx.duplicated(take_last=True)
# Index([True, True, False, False, False, False], dtype='bool')
idx.drop_duplicates(take_last=True)
# Int64Index([3, 4, 1, 2], dtype='int64')

[jreback]

these need to raise if inplace and it's anindex (as they are immutable)

[jreback]

side note - can u audit existing methods in indexOps for using inplace

[sinhrks]

There seems to be no func accepts inplace other than this.

[jreback]

ok, gr8. still I think putting the check on update_inplace might be good

[jreback]

I think better is to have update_inplace in core/base.py that simply raises if its an Index (I think this would be overriden by the update_inplace in core/generic.py and so other sub-classes won't see it

[sinhrks]

Sure. I think adding update_inplace to Index is clearer?

[jreback]

ahh yes, that would be better (though maybe add as a NotIMplemented to OpsMixIn just as a place holder for the abstract methdos)

[jorisvandenbossche]

About the Series or Index, could you do something like in generic.py with substitution of klass name so that only Series or Index shows up in the respective docstring?

[sinhrks]

@jreback, @jorisvandenbossche Considering both comments and fixed.

Defining update_inplace in IndexOpsMixin results in Series to refer it because of inheritance order. Thus, functions which uses update_inplace must be overridden eventually.
https://github.com/pydata/pandas/blob/master/pandas/core/series.py#L77

So defined common logic in IndexOpsMixin and override in Index and Series each. As a result, Index no longer need update_inplace because it will not accept inplace keyword.

[jreback]

this is very awkward to do. Maybe just put the immutable definition in base and override the definition in series. prob simpler?

[sinhrks]

OK, fixed to centralize the logic to IndexOpsMixin. Even though update_inplace is defined in both IndexOpsMixin and Index, it will never called in drop_duplicates case (Index.drop_duplicates blocks inplace kw, and it is better for proper docstring)

[jorisvandenbossche]

Just as a usage question, what do we envisage as the 'recommended' way to drop duplicate indices from a DataFrame (where you now had to say the somewhat unintuitive df.groupby(level=0).first())?:

df[~df.index.duplicated()]

or

df.reindex(df.index.drop_duplicates())

although these are even longer than the groupby ..

[jreback]

unchanged, the first is best (this is for the Index to be compatible)

[sinhrks]

@jorisvandenbossche 's point is #2825. Though I feel df[~df.index.duplicated()] is simple enough.

[jreback]

the doc string for Series.duplicated/drop_duplicated does not have inplace?

I would change this around. Why don't you just have _duplicated/_drop_duplicates in Base (w/o the inplace argument, and are private (no doc strings)).

Then in Index/Series put in the doc-strings (and inplace for Series)?

[sinhrks]

No, Series has. I expect docstrings are rendered as expected.

Agreed to remove docstring from IndexOpsMixin, and I feel no need to make them private (little confusing).

[jreback]

ok, that's fine then. ping hwne ready

[sinhrks]

Thanks to confirm. Now green.