Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Rolling apply on DataFrame with Datetime index returns NaN #15305

Closed
cafischer opened this issue Feb 4, 2017 · 2 comments · Fixed by #17156
Closed

Rolling apply on DataFrame with Datetime index returns NaN #15305

cafischer opened this issue Feb 4, 2017 · 2 comments · Fixed by #17156
Labels
Bug Datetime Datetime data dtype Reshaping Concat, Merge/Join, Stack/Unstack, Explode
Milestone

Comments

@cafischer
Copy link

Code Sample, a copy-pastable example if possible

import pandas as pd

df = pd.DataFrame({'A': [0, 1, 2, 3, 4]},
                  index=[pd.Timestamp('20130101 09:00:00'),
                         pd.Timestamp('20130101 09:00:01'),
                         pd.Timestamp('20130101 09:00:02'),
                         pd.Timestamp('20130101 09:00:03'),
                         pd.Timestamp('20130101 09:00:04')])
rolled = df.rolling('3s', 3)

print rolled.sum()
print rolled.apply(sum)

outputs

                       A
2013-01-01 09:00:00  NaN
2013-01-01 09:00:01  NaN
2013-01-01 09:00:02  3.0
2013-01-01 09:00:03  6.0
2013-01-01 09:00:04  9.0

                      A
2013-01-01 09:00:00 NaN
2013-01-01 09:00:01 NaN
2013-01-01 09:00:02 NaN
2013-01-01 09:00:03 NaN
2013-01-01 09:00:04 NaN

Problem description

rolled.apply.sum() should give the same output as rolled.apply(sum) but instead only returns nan.

Expected Output

same as rolled.sum()

Output of pd.show_versions()

INSTALLED VERSIONS ------------------ commit: None python: 2.7.12.final.0 python-bits: 64 OS: Linux OS-release: 4.4.0-59-generic machine: x86_64 processor: x86_64 byteorder: little LC_ALL: None LANG: en_US.UTF-8 LOCALE: None.None

pandas: 0.19.2
nose: None
pip: 9.0.1
setuptools: 20.7.0
Cython: None
numpy: 1.12.0
scipy: 0.18.1
statsmodels: 0.6.1
xarray: None
IPython: 5.1.0
sphinx: None
patsy: 0.4.1
dateutil: 2.6.0
pytz: 2016.10
blosc: None
bottleneck: None
tables: None
numexpr: None
matplotlib: 1.5.3
openpyxl: None
xlrd: None
xlwt: None
xlsxwriter: None
lxml: None
bs4: None
html5lib: None
httplib2: None
apiclient: None
sqlalchemy: None
pymysql: None
psycopg2: None
jinja2: None
boto: 2.38.0
pandas_datareader: None

@jreback
Copy link
Contributor

jreback commented Feb 4, 2017

well .apply rolling on time-specified windows isn't tested very well :<

The issue is that the counts computation is using the fixed interval one (see a few lines above). Instead it needs to bascially call counts = roll_count(...) more or les.

see https://github.com/pandas-dev/pandas/blob/master/pandas/window.pyx#L1362

pull-requests are welcome!.

note needs testing with multiple min_periods as well.

@jreback jreback added Bug Difficulty Intermediate Reshaping Concat, Merge/Join, Stack/Unstack, Explode Datetime Datetime data dtype labels Feb 4, 2017
@jreback jreback added this to the 0.20.0 milestone Feb 4, 2017
@jreback jreback modified the milestones: 0.20.0, Next Major Release Mar 23, 2017
@staftermath
Copy link
Contributor

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Bug Datetime Datetime data dtype Reshaping Concat, Merge/Join, Stack/Unstack, Explode
Projects
None yet
Development

Successfully merging a pull request may close this issue.

3 participants