Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

describe casts all values to str for several types #9900

Open
bdice opened this issue Dec 14, 2021 · 3 comments
Open

describe casts all values to str for several types #9900

bdice opened this issue Dec 14, 2021 · 3 comments
Labels
bug Something isn't working Python Affects Python cuDF API.

Comments

@bdice
Copy link
Contributor

bdice commented Dec 14, 2021

The describe implementations, especially for datetime and timedelta types, appear to be casting all the values to str (aside from the numeric implementation). This does not align with Pandas behavior:

>>> s = pd.Series([
...   np.datetime64("2000-01-01"),
...   np.datetime64("2010-01-01"),
...   np.datetime64("2010-01-01"),
... ])
>>> print(type(s.describe()["top"]))
<class 'pandas._libs.tslibs.timestamps.Timestamp'>

I recognize there is an issue here with different types, namely that count and freq are not of the same type as mean, min, percentiles, or max. This also affects numerical columns which will upcast integer values like count to floating types.

Some options to resolve this (and their downsides):

  1. Current implementation: return all values as str (results are on GPU ...but data is not usable as str type).
  2. Return a pd.DataFrame or dict that can have multiple types (not a GPU DataFrame).

I propose changing behavior to adopt option (2), and return a pd.DataFrame. The summary doesn't really need to be a GPU DataFrame since it contains so few values. (Do we have precedent for this kind of behavior returning a CPU (Pandas) DataFrame?)

Originally posted by @bdice in #9867 (comment)

@bdice
Copy link
Contributor Author

bdice commented Dec 14, 2021

Quoting @vyasr from #9867 (comment):

I agree that a copy to host that preserves more type information is probably preferable. I'm not aware of any API that returns a pandas DataFrame, but we do return pandas objects in some places. One example is DataFrame.columns, which returns a pandas Index, probably based on the same consideration of not wanting to put the data on device. Personally from an API design perspective I would prefer not to return pandas objects and would rather return something like a dict, but I can see that causing more friction for users switching between pandas and cudf.

@bdice bdice added Python Affects Python cuDF API. proposal Change current process or code labels Dec 14, 2021
@bdice bdice added bug Something isn't working and removed proposal Change current process or code labels Dec 14, 2021
@github-actions
Copy link

This issue has been labeled inactive-30d due to no recent activity in the past 30 days. Please close this issue if no further response or action is needed. Otherwise, please respond with a comment indicating any updates or changes to the original issue and/or confirm this issue still needs to be addressed. This issue will be labeled inactive-90d if there is no activity in the next 60 days.

@github-actions
Copy link

This issue has been labeled inactive-90d due to no recent activity in the past 90 days. Please close this issue if no further response or action is needed. Otherwise, please respond with a comment indicating any updates or changes to the original issue and/or confirm this issue still needs to be addressed.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug Something isn't working Python Affects Python cuDF API.
Projects
Status: Todo
Development

No branches or pull requests

3 participants