DEPR: Deprecate returning a DataFrame in SeriesApply.apply_standard

[topper-123]

• closes DEPR: Deprecate returning a DataFrame in Series.apply #52116
• Tests added and passed if fixing a bug or adding a new feature
• All code checks passed.
• Added type annotations to new arguments/methods/functions.
• Added an entry in the latest doc/source/whatsnew/vX.X.X.rst file if fixing a bug or adding a new feature.

[rhshadrach]

It's not clear to me what the output in this case will be after the deprecation. Can you give an example?

[topper-123]

It's not clear to me what the output in this case will be after the deprecation. Can you give an example?

This pattern should not be used with Series.apply, but should the user find other ways to archive the same result. So the result after the change will generally look strange using apply.

IF they use the same method after the change, they will get a Series with Series as its elements:

>>> import pandas as pd
>>> ser = pd.Series(range(3))
>>> func = lambda x: pd.Series({"x": x, "x**2": x**2})
>>> ser.apply(func)  # now
   x  x**2
0  1     1
1  2     4
2  3     9
>>> ser.apply(func)  # after change
0    x       0
x**2    0
dtype: int64
1    x       1
x**2    1
dtype: int64
2    x       2
x**2    4
dtype: int64
dtype: object
>>> ser.apply(func)[0]  # single element after change
x       0
x**2    0
dtype: int64

This pattern will be not useful generally after the change is implemented, so they should use better patterns, e.g.

>>> func = lambda x: pd.DataFrame({"x": x, "x**2": x**2})
>>> ser.pipe(func)  # now  & after this deprecation, it's faster
   x  x**2
0  0     0
1  1     1
2  2     4
>>> pd.DataFrame({"x": ser, "x**2": ser**2})  # or this...
   x  x**2
0  0     0
1  1     1
2  2     4

Just find something better/faster than that old pattern...

[topper-123]

I've addressed the comments by @rhshadrach.

[rhshadrach]

lgtm

[rhshadrach]

Thanks @topper-123

[Dr-Irv]

@topper-123 This tripped up a test we do on nightly builds in pandas-stubs, which we can fix, but I think the error message could be improved. If the recommendation is to use Series.pipe() instead to get the same result, then I think that should go in the whatsnew document, i.e., instead of saying "This pattern was very slow and it's recommended to use alternative methods to archive the same goal", say "This pattern was very slow and it's recommended to useSeries.pipe() to achieve the same goal". In addition, including that text about Series.pipe() should be in the error message (assuming that's the preferred workaround).

Also, the error message doesn't look right due to spacing:

>>> import pandas as pd
>>> pd.__version__
'2.1.0.dev0+463.ge9e034badb'
>>> s = pd.Series([-10, 2, 2, 3.4, 10, 10])
>>> def makeseries(x: float) -> pd.Series:
...     return pd.Series([x, 2 * x])
...
>>> s.apply(makeseries)
<stdin>:1: FutureWarning: Returning a DataFrame from Series.apply when the supplied functionreturns a Series is deprecated and will be removed in a future version.
      0     1
0 -10.0 -20.0
1   2.0   4.0
2   2.0   4.0
3   3.4   6.8
4  10.0  20.0
5  10.0  20.0

Note the "functionreturns" misses a space.

Let me know if I should create a separate issue for this.

[topper-123]

Hi @Dr-Irv.

The ability to have series put into a DataFrame can be misused in many different manners. That pipe example was a way to be able to do that particular operation vectorized, i.e. fast. and it can't be generalized.

A general pattern you could exchange the deprecated patterns with would be to do it like ser.pipe(lambda x: pd.DataFrame(x.map(value_func_that_returns_series).to_list())) though that would be dog slow also. To me that's a quite ugly expression and I guess we could inform them about it in the whatsnew section, with a clear warning about performance and that users should find other patterns to do the operation. IMO that code pattern should not be in the error message, it could encourage slow code.,

You're right about the error message, there's a missing space there.

Tell me what you think about the above.

[Dr-Irv]

Tell me what you think about the above.

I think you'll need to expand on this topic in the whatsnew document. Especially since you removed this from the userguide:

``apply`` on a Series can operate on a returned value from the applied function
that is itself a series, and possibly upcast the result to a DataFrame:

.. ipython:: python
    def f(x):
        return pd.Series([x, x ** 2], index=["x", "x^2"])
    s = pd.Series(np.random.rand(5))
    s
    s.apply(f)

This is a useful construct - you have a Series, and you want to get a DataFrame that has multiple columns that are functions of the Series. Now maybe the recommended way of changing the above code would be:

pd.DataFrame(s, columns=['s']).assign(x=lambda df: df.s, x2=lambda df: df.s ** 2).drop(columns=["s"])

There are clearly multiple alternatives, but with this deprecation, I think you might want to demonstrate those alternatives in the whatsnew document, since we had it in the user guide before as a pattern that people most likely copied.

[phofl]

This message should point users to a resource that is explaining what to do now

[phofl]

Agreed with @Dr-Irv

This needs more explanation somewhere about what to do now

[phofl]

There isa DeprecationWarning missing:

df = pd.DataFrame({"x": [1, 2, 3, 4], "y": [10, 20, 30, 40]})


def return_df(x):
    # will create new DataFrame which columns is ['sum', 'mean']
    return pd.Series([x.sum(), x.mean()], index=["sum", "mean"])

df.apply(return_df, axis=1)

Looks like axis takes another path?