ENH: Implement multi-column DataFrame.quantiles

[charlesbluca]

Rough attempt at implementing cuDF's DataFrame.quantiles; shares a lot of common logic with sort_values, as the indexer that sorts the dataframe by all columns is ultimately what is used to grab the desired quantiles.

cc @quasiben @rjzamora

• closes ENH: Add support for multi-column quantiles of DataFrame #43881
• tests added / passed
• Ensure all linting tests pass, see here for how to run them
• whatsnew entry

[quasiben]

@TomAugspurger this is an attempt at trying to get multi-column sorting working in Dask which requires a multi-column quantile

[rjzamora]

cudf's DataFrame.quantiles doesn't support a numeric_only argument, so the effective default is numeric_only=False. Any chance we could modify the default here? Is this meant to align with quantile arguments?

Note that I can understand the argument for numeric_only=True, but it may add a bit of extra pain in Dask :)

[charlesbluca]

Yeah, I did this to align with the default for quantile - happy to change to False if it make sense to the devs

[TomAugspurger]

I'm not sure about the name... I worry about having both a .quantile and quantiles, and I don't think that quantiles really conveys what's different about this method. To me the plural version sounds like you're able to compute multiple quantiles at once, which is already supported by passing a list to quantile.

So if we do this, I'd suggest a name like multi_column_quantile or multi_quantile or quantile_table (the last one mirrors a proposed API for groupby.agg on a table). Or perhaps we make this a keyword for quantile, like table=True or columnar=False? Because IIUC the output type (Series or Frame) and labels will be identical .quantile, the only difference is the values?

[quasiben]

@shwina, do you think we could alias quantiles in cuDF if we do end up changing the name here ?

[shwina]

Yes -- that should be OK for cuDF. I also like multi_quantile or quantile_table over quantiles

[charlesbluca]

Cool! In that case I'm going to rename this method multi_quantile, and we can follow up with a rename / aliasing on cuDF?

[jreback]

@charlesbluca this hasn't received any scruity yet. -1 on adding methods directly like this.

[jreback]

tests are the first thing that is needed

[rjzamora]

-1 on adding methods directly like this.

@jreback - Is your preference to add a new option to the existing quantile method. Or are you saying that down-stream libraries should be implementing logic like this themselves?

[jreback]

i would add the argument method='single|table' which is what we do for .rolling and friends

[jreback]

this should all be in the quantile method in algos not here

[charlesbluca]

For clarity, do you mean that this code should be in a new quantile method in algos that handles the table case, or in BlockManager.quantile, where it looks like the internal implementation of DataFrame.quantile resides?

[charlesbluca]

Currently blocked on handling for sparse arrays - I am using BlockManager.take to perform the actual quantile operation, which retains sparse dtypes where BlockManager.quantile would not. This causes issues when handling non-list-like qs as an iloc across columns is required, which does not seem to be possible with sparse dtypes:

    def test_quantile_sparse(self, df, expected):
        # GH#17198
        # GH#24600
>       result = df.quantile(interpolation="nearest", method="table")

pandas/tests/frame/methods/test_quantile.py:591: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
pandas/core/frame.py:10475: in quantile
    return res.iloc[0]
pandas/core/indexing.py:957: in __getitem__
    return self._getitem_axis(maybe_callable, axis=axis)
pandas/core/indexing.py:1509: in _getitem_axis
    return self.obj._ixs(key, axis=axis)
pandas/core/frame.py:3483: in _ixs
    new_values = self._mgr.fast_xs(i)
pandas/core/internals/managers.py:974: in fast_xs
    result = cls._empty((n,), dtype=dtype)
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _

cls = <class 'pandas.core.arrays.sparse.array.SparseArray'>, shape = (2,), dtype = Sparse[int64, 0]

    @classmethod
    def _empty(cls, shape: Shape, dtype: ExtensionDtype):
        """
        Create an ExtensionArray with the given shape and dtype.
        """
        obj = cls._from_sequence([], dtype=dtype)
    
        taker = np.broadcast_to(np.intp(-1), shape)
        result = obj.take(taker, allow_fill=True)
        if not isinstance(result, cls) or dtype != result.dtype:
>           raise NotImplementedError(
                f"Default 'empty' implementation is invalid for dtype='{dtype}'"
            )
E           NotImplementedError: Default 'empty' implementation is invalid for dtype='Sparse[int64, 0]'

pandas/core/arrays/base.py:1492: NotImplementedError

A few follow up questions here:

• Do we want quantile(method="table") to not retain sparse dtypes to match quantile(method="single")?
• If not, are there any blockers to ilocing across sparse columns that can be addressed to unblock this PR?

[github-actions]

This pull request is stale because it has been open for thirty days with no activity. Please update or respond to this comment if you're still interested in working on this.

[charlesbluca]

Still interested in working on this, currently blocked by the handling for sparse arrays, specifically if want to retain sparse dtypes for quantile(method="table")

[charlesbluca]

Thanks for pushing this along @mroeschke 😄 some small questions around the modified tests:

[charlesbluca]

Is this test asserting that the output columns are correct when method == "table"?

[mroeschke]

The result will be Series objects, but I improved the assertions here to compare the entire Series results if the interpolation is linear (& method = single) or compare Series name + index if interpolation is nearest (& method = table)

[charlesbluca]

Is this test asserting that the output columns are correct when method == "table"?

[mroeschke]

The result will be Series objects, but I improved the assertions here to compare the entire Series results if the interpolation is linear (& method = single) or compare Series name + index if interpolation is nearest (& method = table)

[jreback]

maybe i am not getting something, but why isn't this just
np.asarray(res_df).ravel() and then reed to the existing quantile routine?

[mroeschke]

1. I think that approach would not work for DataFrames with mixed dtypes
2. For the limited set of interpolation methods supported (to start) in this PR, I think this approach is more performant as only quantile indices are calculated followed by a take.

[jreback]

thanks @charlesbluca really nice! (and @mroeschke for pushing over the line)