API: _constructor and subclasses

[TomAugspurger]

Followup to #31925.

Over there, we removed _ensure_type because it broke things like

import pandas as pd
import pandas.testing as pdt

class MyDataFrame(pd.DataFrame):
    pass

mdf = MyDataFrame()
df = pd.DataFrame()
# In DataFrame.reindex_like, the derived class is compared to
# the base class. The following line will throw
# 'AssertionError: <class 'pandas.core.frame.DataFrame'>'
mdf.reindex_like(df)

However, I don't think that example is very compelling. It's strange for MyDataFrame.reindex to return anything other than MyDataFrame. The root cause is MyDataFrame._constructor returning DataFrame, rather than MyDataFrame.

I think we should somehow get subclasses to have their _constructor return something that returns their subclass. We can

1. Require this, by doing a runtime check on the DataFrame._constructor
2. Change DataFrame._constructor to return type(self)

2 is nicer, but it requires that the subclasses __init__ method be compatible enough with DataFrame's. It's probably safe to assume that, but it might be an API breaking change.

[jbrockmendel]

If we were to go for option 2, why not just get rid of _constructor and use type(self) in the relevant places? The important ones are _constructor_sliced and _constructor_expanddim

[jorisvandenbossche]

I don't think we need to require this (we also don't require _constructor to be a class, it can be a constructor function).

When having a DataFrame subclass, not every operation in pandas necessarily needs to return that subclass, depending on your use case and characteristics of your subclass.
For example, in GeoPandas, an operation like reindex can remove the geometry column, in which case it makes sense to return a normal DataFrame.

[jorisvandenbossche]

Whether to change the default (item 2), that might make sense (it seems a more sensible default, maybe).
But, that's also a breaking change, so if we want to change that, I would keep that for 2.0.

If we want to do larger changes on how this _constructor mechanism is working, I think we should also seek feedback of more people that are actually using subclasses.

[jbrockmendel]

Another candidate criterion that I'm finding in #33357 would be desirable: make this a class attribute instead of an instance attribute, like Timestamp.min, so that DataFrame._constructor is DataFrame instead of a property object.

Granted, the semantics of implementing that are pretty ugly.

[jbrockmendel]

If we want to do larger changes on how this _constructor mechanism is working, I think we should also seek feedback of more people that are actually using subclasses.

@jorisvandenbossche you mentioned geopandas returns a callable instead of a class. How difficult would it be to change that?

Are we aware of any other downstream libraries that do that?

[jorisvandenbossche]

See my explanation above #32638 (comment), about that this is actually a feature for geopandas that we don't have to return a class.

I suppose I could make a dummy class with a __new__ that has the same behaviour as the callable, but not sure what the value of that would be then.

So my question back is: what are the use cases for requiring it to be a class? (apart from the PR with the _init_mgr)

[jbrockmendel]

Just grepping for \._constructor(_sliced|_expanddim)?\. shows that in core.groupby.generic we use self._obj._constructor._from_arrays

[jorisvandenbossche]

That was recently added: #33884