Skip to content

Commit

Permalink
[docs] Add ViLT model checkpoint instructions
Browse files Browse the repository at this point in the history
Add ViLT model checkpoints to model zoo.
Add ViLT finetuning and pretraining instructions
to project docs.

ghstack-source-id: 52b80d03e12eedb1ad7d45c718d5fa48a206e785
Pull Request resolved: #1161
  • Loading branch information
Ryan-Qiyu-Jiang committed Nov 29, 2021
1 parent b672a74 commit 19a6542
Showing 2 changed files with 32 additions and 1 deletion.
16 changes: 16 additions & 0 deletions mmf/configs/zoo/models.yaml
Original file line number Diff line number Diff line change
@@ -546,3 +546,19 @@ unit:
- url: mmf://models/unit_models/unit.all_8_datasets.shared_dec_with_coco_init_without_task_embedding.tar.gz
file_name: unit.all_8_datasets.shared_dec_with_coco_init_without_task_embedding.tar.gz
hashcode: c4a3c1072f07e605de76402ff7c0e2bc8d4514561c52dc94b3d28b3d4d7f698e

vilt:
defaults: ${vilt.vqa}
vqa:
# Model from project : projects/vilt, Val: 69.55
version: 1.0_2021_011_29
resources:
- url: mmf://models/vilt/vilt.finetuned.vqa2.tar.gz
file_name: vilt.finetuned.vqa2.tar.gz
hashcode: 83539fb58f19592f4286a499557c40f084f366722700232943824922160aeacd
pretrained:
version: 1.0_2021_011_29
resources:
- url: mmf://models/vilt/vilt.pretrained.tar.gz
file_name: vilt.pretrained.tar.gz
hashcode: a933ef78e9afafe044de22ef042d4ffe50388c19544921db044cd446e688c5fd
17 changes: 16 additions & 1 deletion website/docs/projects/vilt.md
Original file line number Diff line number Diff line change
@@ -24,9 +24,24 @@ Follow installation instructions in the [documentation](https://mmf.readthedocs.

## Training

To train ViLT model on the VQA2.0 dataset, run the following command
To train ViLT model from scratch on the VQA2.0 dataset, run the following command
```
mmf_run config=projects/vilt/configs/vqa2/defaults.yaml run_type=train_val dataset=vqa2 model=vilt
```

To finetune using different pretrained starting weights, change the `pretrained_model_name` under image_encoder in the config yaml to reference a huggingface model.

To finetrain a pretrained ViLT model on the VQA2.0 dataset,
```
mmf_run config=projects/vilt/configs/vqa2/defaults.yaml run_type=train_val dataset=vqa2 model=vilt checkpoint.resume_zoo=vilt.pretrained
```

To test a ViLT model already finetuned on the VQA2.0 dataset,
```
mmf_run config=projects/vilt/configs/vqa2/defaults.yaml run_type=val dataset=vqa2 model=vilt checkpoint.resume_zoo=vilt.vqa
```

To pretrain a ViLT model from scratch on the COCO dataset,
```
mmf_run config=projects/vilt/configs/masked_coco/pretrain.yaml run_type=train_val dataset=masked_coco model=vilt
```

0 comments on commit 19a6542

Please sign in to comment.