More Training Information on Reacher

[ivo-1]

Hi,

first and foremost thanks a lot for releasing the code and this great work in general!

While you report great success on many games, we were wondering if you could release more information about your results on the Reacher dataset, namely:

1. Maximum episode return available in the dataset for the Reacher medium-replay dataset (just like in the other plots in figure 4 of the paper)
2. Your loss with the Decision Transformer for any or all of the Reacher datasets, just like you posted in Do both BC and DT fit the training data well? #2

Finally, we were wondering if you have any results on an "Expert"-only dataset for any of the games with the Decision Transformer. Per (Rashidinejad et al., 2021)[1]: "imitation learning [...] is suitable for expert datasets and vanilla offline RL [...] often requires uniform coverage datasets" - we think would be interesting to see if this notion applies to the Decision Transformer as well.

Note that we are mainly asking this because we are currently in the progress of trying to apply the Decision Transformer for a simplified version of Bomberman, which notably is a multi-agent environment. We were also encouraged by your comment on issue #14 that this shouldn't be an issue (in theory). However we are converging to a loss of about 0.75, which is obviously not great and indeed doesn't match the performance of the agents' ("expert" because they are strong rule-based agents) policy used to create our own trajectory ("expert"-only) dataset. In fact, the model is about 1/10th as good when playing against three agents that operate on the same policy used to collect the data and is far off from the maximum episode return available in our dataset.

[1]: Paria Rashidinejad, Banghua Zhu, Cong Ma, Jiantao Jiao, and Stuart Russell. 2021.
Bridging offline reinforcement learning and imitation learning: A tale of pessimism.
CoRR, abs/2103.12021