Why "timesteps = horizon_length * max_epochs" ?

[berttggg]

In skrl examples, the timestep are defined by the above equation, but why?

I thought is

timesteps = horizon_length * num_actors * max_epochs

Thank you in advance.

[Toni-SM]

Hi @berttggg

As indicated in the skrl documentation, the training configuration is mapped as far as possible from the rl_games configuration.

In rl_games, the training duration is basically defined by the max_epochs and horizon_length variables as shown (in an adapted and simplified form) in the following code:

while True:
    epoch_num += 1
    for n in range(self.horizon_length):
        ... # collect data (rollouts)
   ... # train the agent
   if epoch_num >= self.max_epochs:
       break

Reference:

• https://github.com/Denys88/rl_games/blob/990b4782ad0375652af76266a12753cb11d768c6/rl_games/common/a2c_common.py#L1318
• https://github.com/Denys88/rl_games/blob/990b4782ad0375652af76266a12753cb11d768c6/rl_games/common/a2c_common.py#L672

[berttggg]

Thank you so much so the explanation.

But I am confused.

In rl_games documentation:
timesteps = horizon_length * num_actors * max_epochs

So, the data size is governed by horizon_length * num_actors.

In skrl documentation:
timesteps = horizon_length * max_epochs

In skrl, the num_actors (num_envs) seem to be left out?

[Toni-SM]

Hi @berttggg

rl_games counts (for statistics) the timesteps of each parallel environment (num_actors) independently, while skrl does it for the whole task (where calling env.step(...) executes a timestep for all parallel environments in one go).