Remove internal usage of core.index.as_index in favor of cudf.Index

[mroeschke]

Description

cudf.Index.__init__ essentially calls as_index immediately internally. To avoid both from potentially diverging, the public cudf.Index should be preferred to ensure the public behaviors are used internally

Checklist

• I am familiar with the Contributing Guidelines.
• New or existing tests cover these changes.
• The documentation is up to date with these changes.

[vyasr]

There was a time when all Index classes implemented __new__ instead of __init__ as a way to allow the parent Index class to construct indexes of the appropriate type. A lot of the class hierarchy was reworked to simplify the constructor logic and ensure that you could still return the right types. pandas 2.0 simplified this pretty substantially by getting rid of all the subclasses of Index corresponding to numerical and string types. Is it still possible for pd.Index(...) to return an object of a type other than Index itself? IIRC we now have to explicitly construct CategoricalIndex, MultiIndex, and RangeIndex, right? The only caveat is the tupleize_cols argument, which throws a bit of a wrench in things; is that planned to remain long-term in pandas?

Basically what I'm wondering is whether we can go even further than this PR currently does (probably in a follow up) to where we remove as_index altogether (or at least make it an internal only helper function used in specific cases; right now I'm sure it gets used externally by projects like cuspatial and cuml) and also reduce the complexity of the index hierarchy without needing to implement __new__, which adds confusion I'd like to avoid if possible.

[mroeschke]

Is it still possible for pd.Index(...) to return an object of a type other than Index itself?

Yeah the pandas.Index constructor can still return the remaining subclasses e.g.

In [1]: import pandas as pd

In [2]: import datetime

In [3]: pd.Index([datetime.datetime.now()])
Out[3]: DatetimeIndex(['2024-06-03 15:29:10.717813'], dtype='datetime64[us]', freq=None)

The only caveat is the tupleize_cols argument, which throws a bit of a wrench in things; is that planned to remain long-term in pandas?

Yeah I imagine so in order to give users a way to construct a pandas.Index of tuple objects instead of performing "inference" and converting the tuples to pandas.MulitIndex levels

Basically what I'm wondering is whether we can go even further than this PR ...

I think once we get other RAPIDS libaries to stop using as_index we can inline it in the cudf.Index constructor, but I think it will have to remain in __new__ (or similar) since that's also how pandas dispatches Index to it's subclasses. I also dislike that pandas.Index can return it's subclasses, but since the int and float subclasses were removed in pandas 2.0 maybe there will be more interest from the core team to have Index just be the type to hold more date types that the subclasses handle

[vyasr]

It would be nice if we could move towards a world where the Index constructor only returned instances of itself. In the meantime, maybe the best that we can do is subclass index with an intermediate class that overrides __new__ and gets rid of the specializing behavior, that way subclasses can just override __init__. That might alleviate some of the complexity that would otherwise be present when overriding a parent class that implements a nontrivial __new__.

[mroeschke]

/merge