This folder contains the implementation of the PLOT++ method which extends PLOT to support the ViT backbones. Models are coming (in two days).
Following CoOp, it is needed to install the 'dassl' environment. You can follow the scripts Dassl to install dassl as well as PyTorch. Then, you can install other package by running pip install -r requirements.txt (this should be done when dassl is activated).
Please follow the instructions DATASETS.md to construct the datasets.
Our PLOT++ includes two stages, both two running scripts are in scripts/.
To run the first stage, you should firstly run cd ./scripts and change your_data_path and your_work_path in scripts/main_visiononly.sh, then you can run the commands bash main_visiononly.sh DATASET M under plot-pp/scripts/. For example bash bash main_visiononly.sh caltech101 4
DATASET takes as input a dataset name, like caltech101.
M is the number of vision prompts, such as 4.
After the first stage, you can find the results from output_visiononly/, whose structure is
output
|–– OP_N4/caltech101/
| |–– PLOTPP/
| | |–– vit_16shots/
| | | |–– nctx4_cscFalse_ctpend/
| | | | |–– seed1/
| | | | |–– seed2/
| | | | |–– seed3/
| | |–– vit_8shots/
| | | |–– nctx4_cscFalse_ctpend/
| | | | |–– seed1/
| | | | |–– seed2/
| | | | |–– seed3/
you can find the first-stage pretrained model from seed1/visiononly/model.pth.tar-50.
To run the second stage, you should change your_data_path, your_pretrain_path and your_work_path in scripts/main_joint.sh, then you can run the commands bash main_joint.sh DATASET M under plot-pp/scripts/. For example bash main_joint.sh caltech101 4
Note that DATASET and M should have the same values as the fist stage.
You can find the results from output_joint/ whose structure is
output_joint
|–– OP_N4/caltech101/
| |–– PLOTPP/
| | |–– vit_16shots/
| | | |–– nctx4_cscFalse_ctpend/
| | | | |–– seed1/
| | | | |–– seed2/
| | | | |–– seed3/
| | |–– vit_8shots/
| | | |–– nctx4_cscFalse_ctpend/
| | | | |–– seed1/
| | | | |–– seed2/
| | | | |–– seed3/
Results reported below show accuracy across 11 recognition datasets averaged over 1 2 4 8 16 shots, you can download corresponding vision-only models (stage1) and joint-train models (stage2) through links given in corresponding columns.
| Dataset | 1shots | model | 2shots | model | 4shots | model | 8shots | model | 16shots | model |
|---|---|---|---|---|---|---|---|---|---|---|
| Caltech101 | 94.34 | link | 94.69 | link | 95.13 | link | 95.51 | link | 96.04 | link |
| DTD | 54.57 | link | 56.72 | link | 62.43 | link | 66.49 | link | 71.43 | link |
| EuroSAT | 65.41 | link | 76.80 | link | 83.21 | link | 88.37 | link | 92.00 | link |
| FGVCAircraft | 28.60 | link | 31.14 | link | 35.29 | link | 41.42 | link | 46.74 | link |
| Flowers102 | 80.48 | link | 89.81 | link | 92.93 | link | 95.44 | link | 97.56 | link |
| FOOD101 | 86.16 | link | 86.33 | link | 86.46 | link | 86.58 | link | 87.11 | link |
| ImageNet | 66.45 | link | 68.28 | link | 70.40 | link | 71.31 | link | 72.60 | link |
| OxfordPets | 91.89 | link | 92.29 | link | 92.55 | link | 93.02 | link | 93.59 | link |
| StanfordCars | 68.81 | link | 73.17 | link | 76.25 | link | 81.26 | link | 84.55 | link |
| SUN397 | 66.77 | link | 68.06 | link | 71.73 | link | 73.93 | link | 76.03 | link |
| UCF101 | 74.31 | link | 76.76 | link | 79.76 | link | 82.80 | link | 85.34 | link |
You can use /scripts/evaluation.sh to evaluate PLOT++ by loading our released models from links shown above, whose structure is
plot++_models
|–– caltech101/
| |–– 1shots/
| | |–– stage1/
| |–– |–– |–– seed1/
| | |–– stage2/
| | |–– |–– seed1/
| | |–– |–– seed2/
| | |–– |–– seed3/
To evaluate the first-stage models, you should firstly gain the stage1 model, and store it in your_pretrain_path, change your_data_path, your_pretrain_path and your_work_path in scripts/evaluation.sh, then you can run the command bash evaluation.sh DATASET 4 under plot-pp/scripts/. For example bash evaluation.sh imagenet 4. To evaluate the stage2 models, you should firstly gain the second-stage model and repeat what you do when evaluating the first-stage model.