model.pretrain() using DDPG+HER

[Cladett]

Hello,

I have tried to use HER+DDPG to pretrain an agent based on some recorded demonstrations.
From the error I obtained i believe right now the library does not offer this feature, correct?
When i pretrain only using DDPG everything run smootly but with her i encounter problems.

Training with HER

  model_class = DDPG  # works also with SAC, DDPG and TD3                                                                                                        
 goal_selection_strategy = 'future' # equivalent to GoalSelectionStrategy.FUTURE                                                                                
 print('Ready to test HER')                                                                                                                                     
 model = HER('MlpPolicy', wrapped_env, model_class, n_sampled_goal=4, goal_selection_strategy=goal_selection_strategy,                                           tensorboard_log="./her_dvrl_tensorboard/" ,verbose=1 )                                                                                                                                                                 
                                                                                                                                                                
 # Pretrain the model using the demonstration                                                                                                                   
  model.pretrain(dataset, n_epochs=1000)                                                                                                                         
  model.save("./models/pretrain_1.12.20", cloudpickle = True)

[Miffyli]

Yes, there is no pretrain in stable-baselines3 (there is some crude support in stable-baselines). See e.g. d3rlpy for offline RL algorithms.

[Cladett]

Thank you for your answer. How about the imitation library in stable-baselines3?
I thought that was the equivalent of model.pretrain() from stable-baselines.

[araffin]

Hello,
You have different solution:

• if you just want to pretrain using behavior cloning, best is probably to write some custom code, we give some examples (see Add support for pretraining [feature request] #27 )
• for behavior cloning, you can also take a look at the imitation repo (cf doc), but I'm not sure if HER is supported
• if you want to use offline learning, then you can take a look at the offline learning repo that @Miffyli mentioned (but I'm also not sure that HER is supported there)

EDIT: note for myself, we need to add this to the migration guide, it is missing currently

[OscarGarciaF]

So there won't be any model.pretrain() in the future?

Why would a feature that was so simple to do on previous versions be missing: https://stable-baselines.readthedocs.io/en/master/guide/pretrain.html

[Miffyli]

So there won't be any model.pretrain() in the future?

Not planned anytime soon as other libraries fill this part. It might be considered later on, however to correctly (and cleanly) support these would require extensive additions and careful implementing, hence we refer to other libraries focusing on those algorithms for the time being.