BUG: agg() function on groupby dataframe changes dtype of datetime64[ns] column to float64 #12992

facaiy · 2016-04-26T12:35:02Z

closes #12821
closes #12941

jreback · 2016-05-07T19:14:06Z

can you rebase / update

facaiy · 2016-05-08T13:02:53Z

Sorry, @jreback.

This month I was far busy than expected, and I'm planning to fix the bug on next Sunday.

codecov-io · 2016-05-12T18:37:14Z

Current coverage is 85.30% (diff: 100%)

Merging #12992 into master will decrease coverage by <.01%

@@             master     #12992   diff @@
==========================================
  Files           139        139          
  Lines         50138      50140     +2   
  Methods           0          0          
  Messages          0          0          
  Branches          0          0          
==========================================
+ Hits          42769      42770     +1   
- Misses         7369       7370     +1   
  Partials          0          0

Powered by Codecov. Last update 3186fef...607a170

jreback · 2016-05-25T16:01:51Z

can you rebase / update?

jreback · 2016-05-27T00:19:14Z

doc/source/whatsnew/v0.18.2.txt

@@ -139,3 +139,5 @@ Bug Fixes
 - Bug in ``NaT`` - ``Period`` raises ``AttributeError`` (:issue:`13071`)
 - Bug in ``Period`` addition raises ``TypeError`` if ``Period`` is on right hand side (:issue:`13069`)
 - Bug in ``pd.set_eng_float_format()`` that would prevent NaN's from formatting (:issue:`11981`)
+
+- Bug in ``agg()`` function on groupby dataframe changes dtype of ``datetime64[ns]`` column to ``float64`` (:issue:`12821`)


Bug in .groupby(..).agg(..) could change the dtype of ......

jreback · 2016-05-27T00:20:21Z

does this also address #12941 ?

pls rebase / small update as well. ping on green.

facaiy · 2016-05-29T07:36:52Z

Hi, I have fixed #12941 and rebase again.

jreback · 2016-05-29T14:52:51Z

pandas/tests/frame/test_timeseries.py

+                                   pd.Timestamp('2012-05-01')]})
+
+        res = df.min()
+        tm.assert_attr_equal('dtype', df['foo'], res)


construct an expected result and use assert_series_equal

jreback · 2016-05-30T14:14:56Z

doc/source/whatsnew/v0.18.2.txt

@@ -318,3 +319,7 @@ Bug Fixes


 - Bug in ``Categorical.remove_unused_categories()`` changes ``.codes`` dtype to platform int (:issue:`13261`)
+
+- Bug in ``agg()`` function on groupby dataframe changes dtype of ``datetime64[ns]`` column to ``float64`` (:issue:`12821`)


Bug in ``.agg()` on a groupby, changes the dtype of .....

jreback · 2016-05-30T14:16:04Z

ok looks good. pls rebase / squash (and build failed).

facaiy · 2016-06-01T12:56:32Z

rebase again.

jreback · 2016-06-30T10:17:16Z

can you rebase and i'll review

jreback · 2016-07-06T22:13:26Z

can you rebase. ping when green.

facaiy · 2016-07-24T10:42:39Z

@jreback, hi, agg function will eat up Name, is it a bug?

jreback · 2016-07-24T10:54:55Z

not sure can u show s complete example

facaiy · 2016-07-24T13:20:19Z

In [12]: df = pd.Series(range(5)).to_frame()

In [13]: df
Out[13]:
   0
0  0
1  1
2  2
3  3
4  4

In [14]: df.min()
Out[14]:
0    0
dtype: int64

In [15]: df.loc[0]
Out[15]:
0    0
Name: 0, dtype: int64

jreback · 2016-07-24T13:43:29Z

Giving a slightly more clear example. I suppose this could be done. Though its only for a very limited number of reducers, prob just min/max. For example, anything that does an aggregation like .sum/.mean this by-definition obliterates the index. Pls make a new issue and list the functions you think should be changed here.

In [19]: df = pd.Series(range(5),index=list('abcde')).to_frame(name='foo')

In [20]: df
Out[20]: 
   foo
a    0
b    1
c    2
d    3
e    4

In [21]: df.loc['a']
Out[21]: 
foo    0
Name: a, dtype: int64

In [22]: df.min()
Out[22]: 
foo    0
dtype: int64

facaiy · 2016-07-28T21:03:24Z

@jreback Hi, I'll submit a issue later. And all checks have passed, so do you think that the pr is ok?

jreback · 2016-07-28T23:48:41Z

pandas/tests/test_groupby.py

+        # test for `first` function
+        exp = df.loc[[0, 3, 4, 6]].set_index('class')
+
+        res = df.groupby('class').first()


need to test

df.groupby('class').time.first() and so on (e.g. repeat with .agg both forms as well)

if making a loop helps to simplify testing, pls do so

facaiy · 2016-07-30T09:00:48Z

@jreback all checks have passed.

jreback · 2016-08-03T15:24:43Z

pandas/tests/frame/test_timeseries.py

+        exp = pd.Series([pd.Timestamp('2012-05-01')], index=["foo"])
+        tm.assert_series_equal(res, exp)
+
+        # GH12941


put a small comment here (1-line) that explains what you are testing

jreback · 2016-08-03T15:25:01Z

lgtm. pls rebase and ping on green.

jreback · 2016-08-03T15:25:44Z

doc/source/whatsnew/v0.19.0.txt

@@ -854,4 +854,9 @@ Bug Fixes
 - Bugs in ``Index.difference`` and ``DataFrame.join`` raise in Python3 when using mixed-integer indexes (:issue:`13432`, :issue:`12814`)

 - Bug in ``.to_excel()`` when DataFrame contains a MultiIndex which contains a label with a NaN value (:issue:`13511`)
+
 - Bug in ``pd.read_csv`` in Python 2.x with non-UTF8 encoded, multi-character separated data (:issue:`3404`)


put bug fixes in blanks spaces (or create them) in-between other fixes to avoid conflicts (IOW, don't put them at the end)

…f datetime64[ns] column to float64

…me64[ns]

facaiy · 2016-08-06T10:37:38Z

@jreback Both #12821 and #12941 are fixed in the PR. The two bugs behave similarity, while they are located in different code. And all checks have passed.

jreback · 2016-08-06T22:57:16Z

thanks!

jreback · 2016-08-06T22:58:01Z

@ningchi can you have a look at these, see if they are also closed (so maybe just need some validation tests).

xref #8617 (might be a duplicate)
xref #4147

facaiy · 2016-08-07T08:57:19Z

Ok

facaiy · 2016-08-13T09:10:15Z

It seems that #8617 has been fixed, while #4147 behaves differently now.

jreback added Bug Datetime Datetime data dtype Groupby labels Apr 26, 2016

facaiy force-pushed the agg_time_dtype branch 2 times, most recently from 40922cb to 1236422 Compare May 12, 2016 15:35

facaiy changed the title ~~WIP: agg() function on groupby dataframe changes dtype of datetime64[ns] column to float64~~ agg() function on groupby dataframe changes dtype of datetime64[ns] column to float64 May 13, 2016

facaiy changed the title ~~agg() function on groupby dataframe changes dtype of datetime64[ns] column to float64~~ BUG: agg() function on groupby dataframe changes dtype of datetime64[ns] column to float64 May 13, 2016

jreback reviewed May 27, 2016
View reviewed changes

jreback added this to the 0.18.2 milestone May 27, 2016

facaiy force-pushed the agg_time_dtype branch from a933d45 to cc2aaa9 Compare May 29, 2016 07:35

jreback reviewed May 29, 2016
View reviewed changes

facaiy force-pushed the agg_time_dtype branch from e0f70b2 to 98b5f59 Compare May 30, 2016 12:45

jreback reviewed May 30, 2016
View reviewed changes

facaiy force-pushed the agg_time_dtype branch from 98b5f59 to 663ded1 Compare June 1, 2016 05:34

facaiy force-pushed the agg_time_dtype branch from 663ded1 to 34fbe1e Compare July 2, 2016 09:03

jreback removed this from the 0.18.2 milestone Jul 6, 2016

facaiy force-pushed the agg_time_dtype branch from 34fbe1e to ed9e2eb Compare July 28, 2016 13:43

jreback reviewed Jul 28, 2016
View reviewed changes

jreback mentioned this pull request Jul 29, 2016

Unexpected behaviour when grouping datetime column containing null-values, SeriesGroupby #10979

Closed

facaiy force-pushed the agg_time_dtype branch from 0885bed to 290ce16 Compare July 30, 2016 07:13

jreback reviewed Aug 3, 2016
View reviewed changes

jreback added this to the 0.19.0 milestone Aug 3, 2016

jreback reviewed Aug 3, 2016
View reviewed changes

facaiy added 4 commits August 6, 2016 11:03

add test function

914ed71

BUG: fix GH12821, agg() function on groupby dataframe changes dtype o…

a949cee

…f datetime64[ns] column to float64

BUG: fix GH12941, Operations on NaT returning float instead of dateti…

8d17eed

…me64[ns]

add whatsnew entry

607a170

facaiy force-pushed the agg_time_dtype branch from 290ce16 to 607a170 Compare August 6, 2016 03:04

jreback closed this in 63e8f68 Aug 6, 2016

facaiy deleted the agg_time_dtype branch August 7, 2016 08:57

jreback mentioned this pull request Aug 15, 2016

BUG: Series.median() treats NaT as INT_MIN #8617

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

BUG: agg() function on groupby dataframe changes dtype of datetime64[ns] column to float64 #12992

BUG: agg() function on groupby dataframe changes dtype of datetime64[ns] column to float64 #12992

facaiy commented Apr 26, 2016 •

edited by jreback

Loading

jreback commented May 7, 2016

facaiy commented May 8, 2016

codecov-io commented May 12, 2016 •

edited

Loading

jreback commented May 25, 2016

jreback May 27, 2016

jreback commented May 27, 2016

facaiy commented May 29, 2016

jreback May 29, 2016

jreback May 30, 2016

jreback commented May 30, 2016

facaiy commented Jun 1, 2016

jreback commented Jun 30, 2016

jreback commented Jul 6, 2016

facaiy commented Jul 24, 2016

jreback commented Jul 24, 2016

facaiy commented Jul 24, 2016

jreback commented Jul 24, 2016

facaiy commented Jul 28, 2016

jreback Jul 28, 2016

facaiy commented Jul 30, 2016

jreback Aug 3, 2016

jreback commented Aug 3, 2016

jreback Aug 3, 2016

facaiy commented Aug 6, 2016

jreback commented Aug 6, 2016

jreback commented Aug 6, 2016 •

edited

Loading

facaiy commented Aug 7, 2016

facaiy commented Aug 13, 2016

		@@ -318,3 +319,7 @@ Bug Fixes


		- Bug in ``Categorical.remove_unused_categories()`` changes ``.codes`` dtype to platform int (:issue:`13261`)

		- Bug in ``agg()`` function on groupby dataframe changes dtype of ``datetime64[ns]`` column to ``float64`` (:issue:`12821`)

BUG: agg() function on groupby dataframe changes dtype of datetime64[ns] column to float64 #12992

BUG: agg() function on groupby dataframe changes dtype of datetime64[ns] column to float64 #12992

Conversation

facaiy commented Apr 26, 2016 • edited by jreback Loading

jreback commented May 7, 2016

facaiy commented May 8, 2016

codecov-io commented May 12, 2016 • edited Loading

Current coverage is 85.30% (diff: 100%)

jreback commented May 25, 2016

jreback May 27, 2016

Choose a reason for hiding this comment

jreback commented May 27, 2016

facaiy commented May 29, 2016

jreback May 29, 2016

Choose a reason for hiding this comment

jreback May 30, 2016

Choose a reason for hiding this comment

jreback commented May 30, 2016

facaiy commented Jun 1, 2016

jreback commented Jun 30, 2016

jreback commented Jul 6, 2016

facaiy commented Jul 24, 2016

jreback commented Jul 24, 2016

facaiy commented Jul 24, 2016

jreback commented Jul 24, 2016

facaiy commented Jul 28, 2016

jreback Jul 28, 2016

Choose a reason for hiding this comment

facaiy commented Jul 30, 2016

jreback Aug 3, 2016

Choose a reason for hiding this comment

jreback commented Aug 3, 2016

jreback Aug 3, 2016

Choose a reason for hiding this comment

facaiy commented Aug 6, 2016

jreback commented Aug 6, 2016

jreback commented Aug 6, 2016 • edited Loading

facaiy commented Aug 7, 2016

facaiy commented Aug 13, 2016

facaiy commented Apr 26, 2016 •

edited by jreback

Loading

codecov-io commented May 12, 2016 •

edited

Loading

jreback commented Aug 6, 2016 •

edited

Loading