Deploy trained policy on C++ appilication

[impala-shen]

Hi,
Thanks for the awesome library.
Now I get a policy trained in isaacgym, and would like to deploy the trained policy on C++. My plan is to load saved checkpoints, and use torch::jit::trace to transfer the dict to torchscipt format, so the policy can be directly import to C++ by libtorch. But I haven't succeed yet. If you have experience or any suggestion on this, please let me know.
Thanks.

[Toni-SM]

Hi @impala-shen and sorry for the late response.

This is a pending implementation.
During the week I will give it a solution.
Please, stay tuned :)

[impala-shen]

Hi
Thanks for the reply. Actually I got further by wrapping the model(
I am using isaacgym environment):

import rl_games.algos_torch.flatten as flatten

class ModelWrapper(torch.nn.Module):
    def __init__(self, model):
        torch.nn.Module.__init__(self)
        self._model = model

    def forward(self, inputs):
        actions, log_prob, outputs = self._model.act({"states": inputs}, 'policy')
        return outputs["mean_actions"]

agent.load("./runs/agent_50000.pt")

example = torch.ones(1, 4)
adapter = flatten.TracingAdapter(ModelWrapper(agent.models["policy"]), example, allow_non_tensor=True)
traced = torch.jit.trace(adapter, adapter.flattened_inputs, check_trace=False)
traced.eval()
output = traced.forward(example)

But when I test the this jit output with the actions output I get in Isaac gym(given the same observations). It doesn't match, even far from each other. So maybe I did sth wrong.
Hope this can help with your solution, looking forward to it.

[Toni-SM]

Hi @impala-shen

I think the problem is that both skrl and rl_games use a preprocessor to standardize the observations. This preprocessor is learned (as if it were just another model) during training.

That is why during the evaluation, the preprocessor must also be included, otherwise, the observations seen by the policy are incorrect.

[impala-shen]

Didn't know about that! thanks

[impala-shen]

Hi,
How is it going? I still have this deployment problem where the jit output with the actions output I get in Isaac gym doesn't match, even with the preprocessor to standardize the observations. Do you have progress on this issue? Any help would be great.
Thanks.

[Toni-SM]

Hi @impala-shen

Can you check if the output after applying the state preprocessor is the same?
You can test the state preprocessor output in Python with the following code (after instantiate the trainer and load the agent checkpoint)

state = ...
preprocessed_state = agent._state_preprocessor(state)

[impala-shen]

Hi
Thanks! It works now

[Toni-SM]

Hi @impala-shen

So, everything is working now?
What was the issue?

[impala-shen]

Hi,
It was because I directly call RunningStandardScaler for preprocessor and got it wrong with loading the parameters.
Now the torchscipt format output the same. Thanks for the help.