NSLP-G

This is repository for NSLP-G. Pytorch implementation of the paper "Non-Autoregressive Sign Language Production with Gaussian Space", BMVC 2021.

Bibtex

If you find this code useful in your research, please cite:

@inproceedings{hwang2021non,
  title={Non-Autoregressive Sign Language Production with Gaussian Space},
  author={Hwang, Eui Jun and Kim, Jung-Ho and Park, Jong C.},
  booktitle={The 32nd British Machine Vision Conference (BMVC 21)},
  year={2021},
  organization={British Machine Vision Conference (BMVC)}
}

Installation 👷

1. Create conda environment

conda env create -f environment.yaml
conda activate nslp

2. Download the datasets

Please download dataset from Phoenix14-T and How2Sign

Note: The data classes are implemented in scripts/data.py, but you can create your own.

How to use NSLP-G

Training Spatial VAE

python scripts/train_spavae.py \
    --accelerator gpu --devices 0 \
    --num_worker 8 --batch_size 64 \
    --dataset_type {DATASET_TYPE} \
    --train_path {TRAIN_PATH} \
    --valid_path {VALID_PATH} \
    --test_path {TEST_PATH}

Training NonAutoregressive SLP

python scripts/train_nslpg.py \
    --accelerator gpu --devices 0 \
    --num_worker 8 --batch_size 64 \
    --vae_ckpt {PRETRAINED_VAE_CKPT} \
    --dataset_type {DATASET_TYPE} \
    --train_path {TRAIN_PATH} \
    --valid_path {VALID_PATH} \
    --test_path {TEST_PATH}

Training Transformer Autoencoder

python scripts/train_tfae.py \
    --accelerator gpu --devices 0 \
    --num_worker 8 --batch_size 64 \
    --dataset_type {DATASET_TYPE} \
    --train_path {TRAIN_PATH} \
    --valid_path {VALID_PATH} \
    --test_path {TEST_PATH}

Evaluation FGD

python scripts/eval_fgd.py \
        --ckpt {CHECKPOINT_PATH}
        --dataset_type {DATASET_TYPE} \
        --train_path {TRAIN_PATH} \
        --valid_path {VALID_PATH} \
        --test_path {TEST_PATH}
        --device {CUDA_or_CPU} \
        --input_dir {INPUT_DIR}