Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

pandas deprecation of Exponential Moving Average function ewma #5

Closed
3 of 5 tasks
rsvp opened this issue Oct 30, 2016 · 5 comments
Closed
3 of 5 tasks

pandas deprecation of Exponential Moving Average function ewma #5

rsvp opened this issue Oct 30, 2016 · 5 comments

Comments

@rsvp
Copy link
Owner

rsvp commented Oct 30, 2016

Description of specific issue

As of pandas 0.18 the use of its ewma() function will produce a warning which looks like:

    FutureWarning: pd.ewm_mean is deprecated for DataFrame and will be 
    removed in a future version, replace with
    DataFrame.ewm(ignore_na=False,span=30.007751938,min_periods=0,adjust=True).mean()

therefore, this deprecation will adversely impact fecon235 exponential moving average
function ema in the yi_1tools module -- when pandas goes to 0.19.

Expected behavior

No warning to be raised.

Observed behavior

Computed result has no errors. The warning is given only the first time when the
exponential moving average function is used.

Strangely, pd.ewm_mean?? produces no results, so the warning is cryptic.

Why would the improvement/fix be useful to most users?

  • Bug prevention for pandas 0.19 and above

Additional helpful details for bugs

  • Problem started recently, but not in older versions
  • Problem happens with all files, not only some files
  • Problem can be reliably reproduced
  • Problem happens randomly
  • fecon235 version: v4.16.0525
  • pandas version: 0.18.0
  • Python version: 2.7.11
@rsvp rsvp added the warning label Oct 30, 2016
@rsvp
Copy link
Owner Author

rsvp commented Oct 30, 2016

This will be a easy fix for all pandas versions since we will not rely on the suggestion
in the warning to use awkward DataFrame.ewm().mean()

Instead we have re-written our function ema in terms of
another function of ours holtlevel as follows:

def ema( y, alpha=0.20 ):
    '''EXPONENTIAL MOVING AVERAGE using traditional weight arg.'''
    #  y could be a dataframe.
    #  ema is mathematically equivalent to holtlevel with beta=0,
    #  thus issue #5 can be easily resolved for all pandas versions.
    return holtlevel( y, alpha, beta=0 )

Exponential moving average is mathematically a special case of
Holt-Winters smoothing where beta is constrained to zero.
The latter is computational more expensive, so this fix
is justified by logic and simplicity.
(Numerically there may be a tiny constant difference
resulting from varied algorithmic treatments of the beginning
of the time-series as the smoothing warms up :)

To emphasize this fact, we have moved our ema() function
from the yi_1tools module to yi_timeseries module.
The move should not pose a problem in our notebooks
since calls of the explicit form yi_1tools.ema() were never used.

This fix shall be effective as of fecon235 v4.16.1030

@rsvp rsvp closed this as completed Oct 30, 2016
rsvp added a commit that referenced this issue Oct 31, 2016
As of pandas 0.18, pd.ewma() is DEPRECATED.
Our old ema() has been commented out in lib/yi_1tools.py
then new ema() has been rewritten in lib/yi_timeseries.py
as a special case of holtlevel()

See issue #5 for details regarding Deprecation warning,
Exponential Moving Average and Holt-Winters smoothing:
#5
@rsvp rsvp added this to the pandas breaks its API milestone Nov 1, 2016
@rsvp
Copy link
Owner Author

rsvp commented Dec 16, 2016

ema() bug fix -- 2016.12.14

The new dependency of ema() on holtlevel() revealed a bug which produced
bizarre exponential moving averages when beta=0.
The details of the CRITICAL fix are here:
35d3783

REMEDY: pull the latest master branch.

rsvp referenced this issue Dec 16, 2016
Modified: yi_timeseries.py
Also a fix for recently rewritten ema() since it assumes beta=0.
The symptoms were bizarre exponential moving average estimates
due to the growth coefficient unintentionally set always to a
non-zero constant equal to y[1]-y[0], rather than zero.
@rsvp
Copy link
Owner Author

rsvp commented Dec 16, 2016

SUMMARY: code for Holt-Winters filter and Exponential moving average

They are bundled in this module:
https://github.com/rsvp/fecon235/blob/master/lib/yi_timeseries.py

ema() is written as a special case of holtlevel()

rsvp added a commit that referenced this issue Dec 25, 2016
See discussion: #5

Include test of holt() and its workout dataframe.
Include test of ema() which is a special case of Holt-Winters.
@Haisam
Copy link

Haisam commented Jan 17, 2018

thanks :P

@rsvp
Copy link
Owner Author

rsvp commented Jun 25, 2018

2018 Addendum for ema()

The fecon235 source code was refactored in https://git.io/fecon236

Here's the specific module for Holt-Winters which includes ema():
https://github.com/MathSci/fecon236/blob/master/fecon236/tsa/holtwinters.py

The code is also useful in finding the optimal parameter used in the
Exponential moving average, given specific data.

And the tests give some numerical examples:
https://github.com/MathSci/fecon236/blob/master/tests/test_holtwinters.py

... requires at least Python 2.7 or Python 3.4.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Projects
None yet
Development

No branches or pull requests

2 participants