Positivity constraint

[TimKreienkamp]

Hi,

great package!

One concern and food for ongoing discussions in the original Prophet was the need to add positivity constraints for external regressors. Is this possible (if only with slight modifications of the pytorch code) here?

Thanks

Tim

[ourownstory]

Hi Tim,

Thank you!

That should be feasible. Do you mind explaining, what kind of positivity constraint you have in mind: preprocessing of the regressor, or only having a positive impact on the target variable?

Best,
Oskar

[TimKreienkamp]

Hi Oskar,

having a positive impact on the target variable. This is a fairly common usecase in Marketing - you might have a regressor like "television spend per day" where you know (from common sense / business / expert knowledge) that it impacts the target (sales or revenue) non-negatively. In the stan-based Prophet implementation, that was apparently a bit a hassle to implement, since you would have to use different (non-negative) prior distributions in the underlying stan code.

I am not too familiar with pytorch - but I assume the regressor effect is implemented with a Dense layer with linear activation (?) In Keras, there is the option to add a "kernel constraint" on the weights, to make them non-negative. Probably something like that exists in pytorch too? If you could point me to the right parts of the code, I could take a stab at it?

[ourownstory]

Joining this with issue #151

[ourownstory]

Let's clarify what @TimKreienkamp and @johnf1004 are looking for:

1. You are interested in modelling an extra regressor, of which domain knowledge dictates that it only has a positive impact on the target variable. Thus, you want the component to be displayed as always positive, and you don't care if the trend is accordingly adjusted downwards. If so, do you want to also set an upward limit or keep that open?
2. Or are you interested in fitting a regressor with both positive/negative impacts (when training), but to filter out the negative contributions to the forecast (when testing/predicting)?
3. You want to constrain the overall forecast to be positive?

Re: 1 & 2: Both lagged regressors and future regressors are of interest?

As both of you pointed out, there are methods to implement this in Pytorch.
Simply adding a Relu could do the job, but it might be good to quickly research if other libraries have used cleverer methods (as Relu clips the gradient for <0). Alternatively, if all input data is positive, the model weights could be constrained to be positive.

[ourownstory]

@TimKreienkamp
Here are the code parts that matter most:

• covariates addition:

  neural_prophet/neuralprophet/forecaster.py

  Line 1063 in 745ffd2

  def add_lagged_regressor(self, name, regularization=None, normalize="auto", only_last_value=False):

• covariates modelling config:

  neural_prophet/neuralprophet/time_net.py

  Line 154 in 745ffd2

  # Covariates

• covariates modelling:

  neural_prophet/neuralprophet/time_net.py

  Line 380 in 745ffd2

  def covariate(self, lags, name):

• future regressors adding:

  neural_prophet/neuralprophet/forecaster.py

  Line 1102 in 745ffd2

  def add_future_regressor(self, name, regularization=None, normalize="auto", mode="additive"):

• future regressors modelling config:

  neural_prophet/neuralprophet/time_net.py

  Line 172 in 745ffd2

  ## Regressors

• future regressors modelling 1:

  neural_prophet/neuralprophet/time_net.py

  Line 463 in 745ffd2

  if "regressors" in inputs:

• future regressors modelling 2:

  neural_prophet/neuralprophet/time_net.py

  Line 346 in 745ffd2

  def scalar_features_effects(self, features, params, indices=None):

[TimKreienkamp]

Thanks Oskar, I'll take a look and report back here!

[mekriti]

Hi Team,
This looks promising.
I'm looking to replicate the add_regressors function from Prophet in neuralprophet and couldn't find it in example notebooks.
Can you please help me with the name of an analogous function ,if at all its possible at the moment?
Thanks

[johnf1004]

Let's clarify what @TimKreienkamp and @johnf1004 are looking for:

1. You are interested in modelling an extra regressor, of which domain knowledge dictates that it only has a positive impact on the target variable. Thus, you want the component to be displayed as always positive, and you don't care if the trend is accordingly adjusted downwards. If so, do you want to also set an upward limit or keep that open?
2. Or are you interested in fitting a regressor with both positive/negative impacts (when training), but to filter out the negative contributions to the forecast (when testing/predicting)?
3. You want to constrain the overall forecast to be positive?

Re: 1 & 2: Both lagged regressors and future regressors are of interest?

As both of you pointed out, there are methods to implement this in Pytorch.
Simply adding a Relu could do the job, but it might be good to quickly research if other libraries have used cleverer methods (as Relu clips the gradient for <0). Alternatively, if all input data is positive, the model weights could be constrained to be positive.

Re 3: Yes, or non-negative specifically. I guess giving the user the option of using Relu would be perfect, or softplus if they want to force >0?

[ourownstory]

looking to replicate the add_regressors function from Prophet in neuralprophet

Hi @mekriti,
We renamed the function to add_future_regressor in order to differentiate it from the new add_lagged_regressor function.
While there is no example notebook as of now, it is described in the documentation.

[ourownstory]

@johnf1004 Yes, a softplus would likely be the best option to force the options to be positive (for a deeper NN), for a linear model, Relu would be better, assuming 0 is a valid value.

However, it would not work on its own due to normalization. Imagine a time series that is supposedly positive e.g. [0,1], but has a few erroneous negative entries of -1, if you use 'minmax' normalization, the model will be fitted on a range of [0,1], where 0 is mapped to 0.5, and the softplus would make sure none of the values are negative. However, once de-normalized, you would still have negative values.
In this case, proper data pre-processing, and/or prediction post-processing max(0, y_hat), would work just fine.

Nevertheless, this could be avoided by throwing a warning/error if a user sets a positivity constraint and then provides negative training values.

Regarding your problem setting in #151, where many values are zero, and positive values are >>0, an additional prediction of the likelihood of the value being non-zero, multiplied by the predicted value, might be appropriate.

If you would like to take a stab at implementing one of these features, I'll be happy to support you in doing so.

[mekriti]

Hi Team, Thanks for the clarification. I'll contribute an example notebook. Please let me know how to proceed. Thanks

…

On Tue, Dec 8, 2020, 10:14 AM Oskar Triebe ***@***.***> wrote: looking to replicate the *add_regressors* function from Prophet in neuralprophet Hi @mekriti <https://github.com/mekriti>, We renamed the function to add_future_regressor in order to differentiate it from the new add_lagged_regressor function. While there is no example notebook as of now, it is described in the documentation <https://ourownstory.github.io/neural_prophet/model/future-regressors/>. — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <https://github.com/ourownstory/neural_prophet/issues/141#issuecomment-740371915>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ABTPQ6YG7E7ZTODCONVUHTDSTWVKXANCNFSM4UKSKGFA> .

[ourownstory]

answered to you in #181 (comment)

[ourownstory]

Converted this issue to a discussion (new github feature), as it seems better suited for this topic.
Added issues #189 and #190

[candalfigomoro]

@ourownstory
For Facebook Prophet, there's a way to force the trend to stay positive (see the "Approach 5" at facebook/prophet#1668). Is it correct that for neural_prophet there is not something like this yet? Thanks

[ourownstory]

Hi @candalfigomoro,
AFAIK ProphetPos has not been merged yet because there was no clearly best approach that was completely stable.
If you would like to experiment with these ideas and see if they might work stable with neuralprophet, feel free to do so and please share your results!