PPO reward normalization works only for default gamma

[Howuhh]

Problem Description

Current implementation of continuous action PPO uses gym.wrappers.NormalizeReward with default gamma value, for all other gamma's except default 0.99 this normalization will be not correct.

cleanrl/cleanrl/ppo_continuous_action.py

Line 92 in 94a685d

env = gym.wrappers.NormalizeReward(env)

Possible Solution

Very easy, just add gamma=args.gamma as an argument to the normalization wrapper.

[Howuhh]

If this is really a problem I will make a fix PR.

[vwxyzjn]

Oh this makes sense! Thanks for raising the issue. I think ppo_procgen.py and ppg_procgen.py also use the reward normalization - feel free to submit a PR to fix them.

[Howuhh]

Fixed.