Fighting CopyCat Agents Behavioral Cloning from Observation Histories

This is the code of the paper Fighting CopyCat Agents Behavioral Cloning from Observation Histories. This code has implemented the most important part of our contribution — Target-Conditioned Adversary module. And it also supports two important baselines: BC-SO and BC-OH.

Dependencies

conda create -n fight-copycat python=3.6
conda activate fight-copycat
pip install -r requirements.txt

Usage

First, generate the dataset (can be skipped because the data has been generated and saved in ./data/trajectories):

python -m imitation_learning.gen_data Hopper 50000

Then normalize the dataset (can be skipped because it has been normalized):

CUDA_VISIBLE_DEVICES=0 python main.py normalize env=Hopper config/hopper.yaml

Behavioral cloning with single observation:

CUDA_VISIBLE_DEVICES=0 python main.py main env=Hopper config/hopper.yaml name=Hopper-bcso policy.policy_mode=bc-so

Behavioral cloning with observation histories:

CUDA_VISIBLE_DEVICES=0 python main.py main env=Hopper config/hopper.yaml name=Hopper-bcoh policy.policy_mode=bc-oh

Our algorithm (FCA):

CUDA_VISIBLE_DEVICES=0 python main.py main env=Hopper config/hopper.yaml name=Hopper-fca policy.policy_mode=fca optim.discriminator_lr=4e-4 policy.gan_loss_weight=2.0

Citations

Please consider citing our paper in your publications if it helps. Here is the bibtex:

@inproceedings{NEURIPS2020_1b113258,
 author = {Wen, Chuan and Lin, Jierui and Darrell, Trevor and Jayaraman, Dinesh and Gao, Yang},
 booktitle = {Advances in Neural Information Processing Systems},
 editor = {H. Larochelle and M. Ranzato and R. Hadsell and M. F. Balcan and H. Lin},
 pages = {2564--2575},
 publisher = {Curran Associates, Inc.},
 title = {Fighting Copycat Agents in Behavioral Cloning from Observation Histories},
 url = {https://proceedings.neurips.cc/paper/2020/file/1b113258af3968aaf3969ca67e744ff8-Paper.pdf},
 volume = {33},
 year = {2020}
}