Keep subclassing in apply #19823

jaumebonet · 2018-02-21T19:14:17Z

When generating new objects on apply, it calls self.obj._constructor and self.obj._constructior_sliced instead of: DataFrame and Series; keeping subclassing.

closes Subclassing is lost when using DataFrame.apply #19822
tests added / passed
passes git diff upstream/master -u -- "*.py" | flake8 --diff
keeps subclass inside DataFrame.apply

When generating new objects from apply, it calls the self.obj._constructor, self.obj._constructior_sliced instead of DataFrame, Series; keeping subclassing.

pep8speaks · 2018-02-21T19:14:19Z

Hello @jaumebonet! Thanks for updating the PR.

Cheers ! There are no PEP8 issues in this Pull Request. 🍻

Comment last updated on February 23, 2018 at 08:49 Hours UTC

chris-b1 · 2018-02-21T21:04:52Z

Thanks - needs some tests, and it looks like it broke something for sparse, otherwise your approach seems reasonable at a glance.

jaumebonet · 2018-02-21T21:31:32Z

Hi @chris-b1,
Thanks!
Yes, I'm trying to pinpoint what exactly has changed... Clearly it has to do with the fact that now it returns SparseSeries instead of Series, but I'm not familiar with SparseSeries, so I'm not sure what difference does that make.

jreback · 2018-02-22T02:10:38Z

can you add some tests in pandas/tests/frame/test_sub_class (and for Series as well)

jreback · 2018-02-22T02:10:53Z

also pls add a release note, other enhancements section is ok

codecov · 2018-02-22T05:54:26Z

Codecov Report

Merging #19823 into master will increase coverage by 0.02%.
The diff coverage is 100%.

@@            Coverage Diff             @@
##           master   #19823      +/-   ##
==========================================
+ Coverage   91.58%   91.61%   +0.02%     
==========================================
  Files         150      150              
  Lines       48908    48906       -2     
==========================================
+ Hits        44792    44804      +12     
+ Misses       4116     4102      -14

Flag	Coverage Δ
#multiple	`89.98% <100%> (+0.02%)`	⬆️
#single	`41.78% <0%> (ø)`	⬆️

Impacted Files	Coverage Δ
pandas/core/apply.py	`96.74% <100%> (-0.04%)`	⬇️
pandas/util/testing.py	`83.85% <0%> (+0.2%)`	⬆️
pandas/plotting/_converter.py	`66.95% <0%> (+1.73%)`	⬆️

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 8768876...87f2c5c. Read the comment docs.

jaumebonet · 2018-02-22T07:27:03Z

Hi @jreback,
I've added a test to check if subclass of Series are kept when using apply on DataFrame.
Series have they own apply function, so the changes do not affect how Series.apply behave.

As per your suggestion, I also added a bullet point in the other enhancement section.

jreback · 2018-02-22T11:26:15Z

doc/source/whatsnew/v0.23.0.txt

@@ -295,6 +295,7 @@ Other Enhancements
 - ``IntervalIndex.astype`` now supports conversions between subtypes when passed an ``IntervalDtype`` (:issue:`19197`)
 - :class:`IntervalIndex` and its associated constructor methods (``from_arrays``, ``from_breaks``, ``from_tuples``) have gained a ``dtype`` parameter (:issue:`19262`)
 - Added :func:`SeriesGroupBy.is_monotonic_increasing` and :func:`SeriesGroupBy.is_monotonic_decreasing` (:issue:`17015`)
+- :func:``DataFrame.apply`` keeps provides the specified ``Series`` subclass when `DataFrame._constructor_sliced`` is defined (:issue:`19822`)


don't use a private property here, just say when Series and ``DataFrame``` are subclassed

jreback · 2018-02-22T11:28:39Z

pandas/tests/frame/test_subclass.py

+            ['Mary', 'Bo', 'weight', 150]],
+            columns=['first', 'last', 'variable', 'value'])
+
+        df.apply(lambda x: check_row_subclass(x))


you need to check the result of this using tm.assert_frame_equal, just use a simple example of an applied function, testing both returning a list, e.g. lambda x: [1, 2, 3] and a sub-classed series

TomAugspurger · 2018-02-22T15:24:48Z

pandas/tests/frame/test_subclass.py

+            [1, 2, 3],
+            [1, 2, 3]])
+
+        expected3 = DesignSeries([[1, 2, 3], [1, 2, 3], [1, 2, 3], [1, 2, 3]])


Looks like this needs to be updated.

jreback · 2018-02-23T02:30:38Z

pandas/tests/frame/test_subclass.py

+        expected3 = tm.SubclassedSeries([
+            [1, 2, 3],
+            [1, 2, 3],
+            [1, 2, 3],


rather than numbering these, can you just do one after the other and use

result = expected =

jorisvandenbossche · 2018-02-23T08:04:13Z

doc/source/whatsnew/v0.23.0.txt

@@ -295,8 +295,10 @@ Other Enhancements
 - ``IntervalIndex.astype`` now supports conversions between subtypes when passed an ``IntervalDtype`` (:issue:`19197`)
 - :class:`IntervalIndex` and its associated constructor methods (``from_arrays``, ``from_breaks``, ``from_tuples``) have gained a ``dtype`` parameter (:issue:`19262`)
 - Added :func:`SeriesGroupBy.is_monotonic_increasing` and :func:`SeriesGroupBy.is_monotonic_decreasing` (:issue:`17015`)
+- :func:``DataFrame.apply`` keeps the specified ``Series`` subclass when ``Series`` and ``DataFrame`` subclasses are defined (:issue:`19822`)


I find this not so clear. It is about the Series type that is passed to the function?

Yes.
When applying a function to a subclassed DataFrame I realised I could not call the specific methods of the subclassed Series that this DataFrame was supossed to generate through _constructor_sliced. As far as I understood, the subclassing of the Series was lost during apply.
With these changes the subclass is now kept inside apply, so method of the expected Series subclass can be used.

OK, can you try to clarify the text a bit? Eg something like "For subclassed DataFrames, DataFrame.apply will now preserve the Series subclass (if defined) when passing the data to the applied function" (but adapt as you like)

sure! I like it as this! :)
will change it!

jreback · 2018-02-24T15:10:37Z

thanks!

Keep subclassing in apply

0df8689

When generating new objects from apply, it calls the self.obj._constructor, self.obj._constructior_sliced instead of DataFrame, Series; keeping subclassing.

apply changes due to tests break

c3c1e5f

FrameColumnApply.fix errors in wrap_result_for_axis

0ef4bcf

jreback added Reshaping Concat, Merge/Join, Stack/Unstack, Explode Compat pandas objects compatability with Numpy or Python functions labels Feb 22, 2018

Adding test and release note

7fd1178

fix typo

021e681

jreback requested changes Feb 22, 2018

View reviewed changes

jaumebonet added 3 commits February 22, 2018 14:21

update test_subclassed_apply and release note

c549397

pep8 styling

956143c

Merge branch 'master' into apply-subclass

dd929a8

TomAugspurger reviewed Feb 22, 2018

View reviewed changes

jaumebonet added 3 commits February 22, 2018 16:30

test fix

06e11d2

fix test

e6bac37

fixed lint "line too long" error in travis ci

e4f1c3e

jreback requested changes Feb 23, 2018

View reviewed changes

jaumebonet added 2 commits February 23, 2018 07:20

fix test styling

8f7a0b4

fixed test typo

87f2c5c

jorisvandenbossche reviewed Feb 23, 2018

View reviewed changes

fix docs

b22e090

jreback added this to the 0.23.0 milestone Feb 24, 2018

jreback approved these changes Feb 24, 2018

View reviewed changes

jreback merged commit 26a2d41 into pandas-dev:master Feb 24, 2018

harisbal pushed a commit to harisbal/pandas that referenced this pull request Feb 28, 2018

Keep subclassing in apply (pandas-dev#19823)

fbc8d72

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Keep subclassing in apply #19823

Keep subclassing in apply #19823

jaumebonet commented Feb 21, 2018 •

edited

Loading

pep8speaks commented Feb 21, 2018 •

edited

Loading

chris-b1 commented Feb 21, 2018

jaumebonet commented Feb 21, 2018

jreback commented Feb 22, 2018

jreback commented Feb 22, 2018

codecov bot commented Feb 22, 2018 •

edited

Loading

jaumebonet commented Feb 22, 2018

jreback Feb 22, 2018

jreback Feb 22, 2018

TomAugspurger Feb 22, 2018

jreback Feb 23, 2018

jorisvandenbossche Feb 23, 2018

jaumebonet Feb 23, 2018

jorisvandenbossche Feb 23, 2018

jaumebonet Feb 23, 2018

jreback commented Feb 24, 2018

Keep subclassing in apply #19823

Keep subclassing in apply #19823

Conversation

jaumebonet commented Feb 21, 2018 • edited Loading

pep8speaks commented Feb 21, 2018 • edited Loading

Comment last updated on February 23, 2018 at 08:49 Hours UTC

chris-b1 commented Feb 21, 2018

jaumebonet commented Feb 21, 2018

jreback commented Feb 22, 2018

jreback commented Feb 22, 2018

codecov bot commented Feb 22, 2018 • edited Loading

Codecov Report

jaumebonet commented Feb 22, 2018

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

jreback commented Feb 24, 2018

jaumebonet commented Feb 21, 2018 •

edited

Loading

pep8speaks commented Feb 21, 2018 •

edited

Loading

codecov bot commented Feb 22, 2018 •

edited

Loading