Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

ROADMAP: add consistent missing values for all dtypes to the roadmap #35208

Merged
merged 3 commits into from
Aug 20, 2020
Merged

ROADMAP: add consistent missing values for all dtypes to the roadmap #35208

Changes from 2 commits
Commits
Show all changes
3 commits
Select commit Hold shift + click to select a range
3a16f7b
ROADMAP: add consistent missing values for all dtypes to the roadmap
jorisvandenbossche Jul 10, 2020
ee62bd0
add notion of different semantics
jorisvandenbossche Aug 7, 2020
7bcb4e6
update with Tom's suggestions
jorisvandenbossche Aug 7, 2020
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
28 changes: 28 additions & 0 deletions doc/source/development/roadmap.rst
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -53,6 +53,34 @@ need to implement certain operations expected by pandas users (for example
the algorithm used in, ``Series.str.upper``). That work may be done outside of
pandas.

Consistent missing value handling
---------------------------------

Currently, pandas has varying missing data interfaces depending on the data
type: pandas uses ``np.nan`` as missing value indicator in floating point data,
``np.nan`` or ``None`` in object dtype data (eg strings, or booleans with
missing values are cast to object), ``pd.NaT`` in datetimelike data. For
categorical or interval data, they return ``np.nan`` on access even when the
categories or intervals are datetime-like. Integer data cannot store missing
data or are cast to float. In addition, ``NaN`` has different semantics as
"nulls" in many other data tools.

Long term, we want to introduce consistent missing value handling accross the
different data types: all data types should support missing values and with the
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Not sure I agree that all data types should support missing values. Non-nullable types could be beneficial

Copy link
Contributor

@TomAugspurger TomAugspurger Jul 13, 2020 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I agree that there's value in pandas ensuring that a column cannot contain NAs. I'm not sure where best to put that invariant: the dtype or the array / column.

But, to sidestep this issue, perhaps something like "pandas should provide consistent missing value handling for all the different kinds of data", i.e. we give you the ability, without saying that every dtype has to be nullable.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I seem to remember that you, Tom and I had a similar discussion in another issues (but can't directly find it).

I agree that the concept of non-nullability is useful/interesting, but as Tom also mentions: non-nullability shouldn't necessarily be a property of the data type itself. Similarly as Tom is working on the "allow_duplicate_labels" flag, we could have "nullable=True/False" flags per column.

Because for the dtype itself to be non-nullable, we should ask ourselves: do we have an example of a data type (that we want to include in pandas) for which it would never be useful to be nullable?

Or to phrase the text in a different way: if a data type supports missing values, it should follow consistent semantics.

Note that right now, pandas doesn't really know the concept of non-nullable. Yes, we have some dtypes that don't support NAs (like integer dtype), but whenever some operation introduces a missing value, we simply upcast to a dtype that can store the missing value (so in practice means changing to float for integer). A proper concept of nullable=False flag wouldn't necessarily work like this, I think.

same behaviour.

To this end, a new experimental ``pd.NA`` scalar that can be used as missing
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Small correction above. Change "that can be used as missing" to "that can be used as a missing"

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Indeed, thanks

value indicator and with a behaviour that deviates from ``np.nan`` has already
been added in pandas 1.0 (and used in the experimental nullable dtypes). Further
work and research is needed to integrate these new semantics with other data
types, and to provide a path forward to make this the default in a future
version of pandas.

This has been discussed at
`github #28095 <https://github.com/pandas-dev/pandas/issues/28095>`__ (and
linked issues), and described in more detail in this
`design doc <https://hackmd.io/@jorisvandenbossche/Sk0wMeAmB>`__.

Apache Arrow interoperability
-----------------------------

Expand Down