Add timeout handling for on-policy algorithms

[araffin]

Description

Motivation and Context

• I have raised an issue to propose this change (required for new features and bug fixes)

closes #633

Types of changes

• Bug fix (non-breaking change which fixes an issue)
• New feature (non-breaking change which adds functionality)
• Breaking change (fix or feature that would cause existing functionality to change)
• Documentation (update in the documentation)

Checklist:

• I've read the CONTRIBUTION guide (required)
• I have updated the changelog accordingly (required).
• My change requires a change to the documentation.
• I have updated the tests accordingly (required for a bug fix or a new feature).
• I have updated the documentation accordingly.
• I have reformatted the code using make format (required)
• I have checked the codestyle using make check-codestyle and make lint (required)
• I have ensured make pytest and make type both pass. (required)
• I have checked that the documentation builds using make doc (required)

Note: You can run most of the checks using make commit-checks.

Note: we are using a maximum length of 127 characters per line

[araffin]

@zhihanyang2022 could you have a look at that one?

[zhihanyang2022]

@araffin Sorry, which part do you want me to take a look? It seems like the changes are passing tests correctly?

[araffin]

@araffin Sorry, which part do you want me to take a look? It seems like the changes are passing tests correctly?

could you review the code? (both logic and style/naming)

[Miffyli]

LGTM with few comments :). I would like second opinion if @zhihanyang2022 has time to look over this (about ~20 lines of relevant code changes).

[zhihanyang2022]

I'm still trying to understand how testing works here, but I've checked the main changes made to on_policy_algorithm.py and test_gae.py and I think they are correct.

Adding gamma * terminal_value to the final reward is logically identical to what's done in OpenAI SpinUp:

https://github.com/openai/spinningup/blob/038665d62d569055401d91856abb287263096178/spinup/algos/pytorch/ppo/ppo.py#L59

[araffin]

I'm still trying to understand how testing works here,

to give a bit more context, I created an environment where I know in advance the true value of each state (because the reward does not depend on the policy) but that is different if you look at it as an infinite or finite horizon problem.
The env is composed of 4 states (0, 1, 2, 3) that are repeated over time, depending on the max number of steps.
For a max episode length of 8, the agent will go into states 0, 1, 2, 3, 0, 1, 2, 3 and receive a reward of one for each state.
Because the remaining time is not included, we break Markov assumption and the value of the first state V(s=0) is not really well defined, it is something between 8 (discounted sum of 8 steps) and 3... (discounted sum of 4 steps).

On the other hand, if we treat the problem as infinite horizon, the true value is a geometric series 1 + 1**gamma + 1**gamma^2 + ... = 1 / (1 - gamma) (we get a reward of one at each step and the value is the discounted sum of it on an infinite horizon)