[Bug][Regression] df.groupby.apply fails under specific conditions

[harmbuisman]

Code Sample, a copy-pastable example if possible

import pandas as pd
import datetime

def get_vals(x):
    return pd.Series([0,1,2], index=[2000, 2001, 2002])

b = list(range(0,3))*2
y = list(range(2000,2003))*2
df = pd.DataFrame({'b':b,'y':y}) 
df['date'] = pd.to_datetime(df['y'].apply(lambda x: datetime.date(x, 1, 1)))


print(pd.__version__)

print(df)

df.groupby(['b']).apply(lambda x: get_vals(x))

Problem description

The above code gives an error (KeyError: 0) in pandas 0.25.1, while it runs as expected in pandas 0.24.2

The apply function returns a timeseries.
I found that triggering the error is related to having a datetime column in the dataframe. Without that column it does not throw an error.

Expected Output

No error and the following output, as can be verified with pandas version 0.24.2:

Output of pd.show_versions()

[paste the output of pd.show_versions() here below this line]
Environment of 0.25.1 that I tested on:

INSTALLED VERSIONS

commit : None
python : 3.7.4.final.0
python-bits : 64
OS : Windows
OS-release : 10
machine : AMD64
processor : Intel64 Family 6 Model 142 Stepping 10, GenuineIntel
byteorder : little
LC_ALL : None
LANG : None
LOCALE : None.None

pandas : 0.25.1
numpy : 1.16.4
pytz : 2019.2
dateutil : 2.8.0
pip : 19.2.2
setuptools : 41.0.1
Cython : None
pytest : None
hypothesis : None
sphinx : None
blosc : None
feather : None
xlsxwriter : None
lxml.etree : None
html5lib : None
pymysql : None
psycopg2 : 2.7.6.1 (dt dec pq3 ext lo64)
jinja2 : 2.10.1
IPython : 7.7.0
pandas_datareader: None
bs4 : None
bottleneck : 1.2.1
fastparquet : None
gcsfs : None
lxml.etree : None
matplotlib : 3.1.1
numexpr : None
odfpy : None
openpyxl : None
pandas_gbq : None
pyarrow : None
pytables : None
s3fs : None
scipy : 1.3.1
sqlalchemy : 1.3.7
tables : None
xarray : None
xlrd : None
xlwt : None
xlsxwriter : None

Environment of 0.24.2 that I tested on:
INSTALLED VERSIONS

commit: None
python: 3.7.3.final.0
python-bits: 64
OS: Windows
OS-release: 10
machine: AMD64
processor: Intel64 Family 6 Model 142 Stepping 10, GenuineIntel
byteorder: little
LC_ALL: None
LANG: None
LOCALE: None.None

pandas: 0.24.2
pytest: 5.1.1
pip: 19.2.3
setuptools: 41.2.0
Cython: 0.29.13
numpy: 1.16.4
scipy: 1.3.1
pyarrow: None
xarray: None
IPython: 7.7.0
sphinx: 2.2.0
patsy: 0.5.1
dateutil: 2.8.0
pytz: 2019.2
blosc: None
bottleneck: 1.2.1
tables: 3.5.2
numexpr: 2.7.0
feather: None
matplotlib: 3.1.1
openpyxl: 2.6.3
xlrd: 1.2.0
xlwt: 1.3.0
xlsxwriter: 1.2.0
lxml.etree: 4.4.1
bs4: 4.8.0
html5lib: 1.0.1
sqlalchemy: 1.3.7
pymysql: None
psycopg2: None
jinja2: 2.10.1
s3fs: None
fastparquet: None
pandas_gbq: None
pandas_datareader: 0.7.0
gcsfs: None

[MarcoGorelli]

Slightly simpler example:

import pandas
import datetime

df = pandas.DataFrame({'a': ['foo'], 'b': [datetime.datetime.today()]})
df.groupby('a').apply(lambda x: pandas.Series(['spam'], index=[42]))

KeyError: 0

However, this works:

import pandas
import datetime

df = pandas.DataFrame({'a': ['foo'], 'b': [datetime.date.today()]})
df.groupby('a').apply(lambda x: pandas.Series(['spam'], index=[42]))

and returns

       42
a        
foo  spam

[TomAugspurger]

Thank. For datetime input, it seems like we expect to return datetime output.

cc @jbrockmendel, since this may be an area you're working near recently.