Skip to content

Latest commit

 

History

History
93 lines (72 loc) · 3.98 KB

README.md

File metadata and controls

93 lines (72 loc) · 3.98 KB

SimAug

This repository contains the code and models for the following ECCV'20 paper:

SimAug: Learning Robust Representations from Simulation for Trajectory Prediction
Junwei Liang, Lu Jiang, Alexander Hauptmann

You can find more information at our Project Page.

If you find this code useful in your research then please cite

@inproceedings{liang2020simaug,
  title={SimAug: Learning Robust Representations from Simulation for Trajectory Prediction},
  author={Liang, Junwei and Jiang, Lu and Hauptmann, Alexander},
  booktitle={Proceedings of the European Conference on Computer Vision (ECCV)},
  month = {August},
  year={2020}
}

Introduction

This paper focuses on the problem of predicting pedestrian future trajectories in unseen scenarios and camera views.

This paper studies the problem of predicting future trajectories of people in unseen cameras of novel scenarios and views. We approach this problem through the real-data-free setting in which the model is trained only on 3D simulation data and applied out-of-the-box to a wide variety of real cameras. Checkout our ECCV'20 presentation here.

Dataset

Here we provide the link to the multi-view trajectory dataset for download.

  • Download links: Google Drive / Baidu Pan (提取码: 6sk5)

  • The dataset includes 5628 1920x1080 videos (1407 reconstructed trajectory samples in 4 camera views) with bounding boxes and scene semantic segmentation ground truth.

  • Video file naming: Example: VIRAT_S_040000_03_000221_000372_F_1879_obs12_pred16_cam1.mp4 everything before "_F_" is the video filename from the original VIRAT dataset, i.e., [videoname]_F_[start_frame_idx]_obs12_pred16_cam[camera_idx, 1-4].

Multi-view trajectories reconstructed from the VIRAT dataset.

The SimAug Algorithm


We propose a novel approach to learn robust representation through augmenting the simulation training data such that the representation can better generalize to unseen real-world test data. The key idea is to mix the feature of the hardest camera view with the adversarial feature of the original view. We refer to our method as SimAug. We show that SimAug achieves promising results on three real-world benchmarks using zero real training data, and state-of-the-art performance in the Stanford Drone and the VIRAT/ActEV dataset when using in-domain training data.

Dependencies

  • Python 2/3; TensorFlow-GPU >= 1.15.0

Pretrained Models

You can download pretrained models by running the script bash scripts/download_single_models.sh.

Testing

Instructions for testing pretrained models can be found here.

Qualitative analysis between baseline (blue) and SimAug trained model (orange).

Training new models

Instructions for training new models can be found here.

Preprocessing the SDD/Argoverse Dataset

Instructions for processing these can be found here. We have already provided the preprocessed files in the testing instructions.