Decoupling Common and Unique Representations for Multimodal Self-supervised Learning

PyTorch implementation of DeCUR. *The main branch includes the up-to-date version for ECCV 2024 (oral); for special interests, the decur-old branch backs up the older version. While the core design remains the same, we added additional deformable attention for ConvNet backbones, more experiments, and more analysis.

Pretrained models

Modality	Pretrain dataset	Deformable Attention	Full checkpoint	Backbone only
SAR-MS	SSL4EO-S12	❌	RN50-SAR/MS-ep100	RN50-SAR, RN50-MS
SAR-MS	SSL4EO-S12	✅	RN50-RDA-SAR/MS-ep100	RN50-RDA-SAR, RN50-RDA-MS
SAR-MS	SSL4EO-S12	❌	ViTS16-SAR/MS-ep100	ViTS16-SAR, ViTS16-MS
RGB-DEM	GeoNRW*	❌	RN50-RGB/DEM-ep100	RN50-RGB, RN50-DEM
RGB-DEM	GeoNRW*	✅	RN50-RDA-RGB/DEM-ep100	RN50-RDA-RGB, RN50-RDA-DEM
RGB-DEM	GeoNRW*	❌	ViTS16-RGB/DEM-ep100	ViTS16-RGB, ViTS16-DEM
RGB-depth	SUNRGBD*	❌	MiTB2-RGB/HHA-ep200	MiTB2-RGB, MiTB2-HHA
RGB-depth	SUNRGBD*	❌	MiTB5-RGB/HHA-ep200	MiTB5-RGB, MiTB5-HHA

*Performance gain when transferring these RGB-DEM/RGB-depth models to other very different downstream datasets may not be significant without additional care, as the pretraining datasets were designed for supervised tasks and are limited in scale and diversity.

DeCUR Pretraining

Clone the repository and install the dependencies based on requirements.txt. Customize your multimodal dataset and your preferred model backbone in src/datasets/, src/models/ and src/pretrain_mm.py, and run

python pretrain_mm.py \
--dataset YOUR_DATASET \
--method PRETRAIN_METHOD \
--data1 /path/to/modality1 \
--data2 /path/to/modality2 \
--mode MODAL1 MODAL2 \
...

Apart from DeCUR, we also support multimodal pretraining with SimCLR, CLIP, BarlowTwins and VICReg.

If you are using distributed training with slurm, we provide some example job submission scripts in src/scripts/pretrain.

Transfer Learning

See the corresponding readme.md in the datasets folders of the SAR-optical/RGB-DEM transfer learning tasks for dataset preparation instructions. To be updated.

Multilabel scene classification with ResNet50 on BigEarthNet-MM:

$ cd src/transfer_classification_BE
$ python linear_BE_resnet.py --backbone resnet50 --mode s1 s2 --pretrained /path/to/pretrained_weights ...

Semantic segmentation with simple FCN on GeoNRW:

$ cd src/transfer_segmentation_GEONRW
$ python GeoNRW_MM_FCN_RN50.py --backbone resnet50 --mode RGB DSM mask --pretrained /path/to/pretrained_weights ...

Semantic segmentation with CMX on SUNRGBD and NYUDv2:

$ cd src/transfer_segmentation_SUNRGBD
$ python convert_weights.py # convert pretrained weights to CMX format

Then refer to https://github.com/huaaaliu/RGBX_Semantic_Segmentation for dataset preparation, training etc.
Simply load the pretrained weights from our pretrained models. 

License

This project is licensed under the Apache 2.0 License - see the LICENSE file for details.

Citation

% for this version
@article{wang2024decoupling,
  title={Decoupling Common and Unique Representations for Multimodal Self-supervised Learning},
  author={Wang, Yi and Albrecht, Conrad M and Braham, Nassim Ait Ali and Liu, Chenying and Xiong, Zhitong and Zhu, Xiao Xiang},
  journal={arXiv preprint arXiv:2309.05300},
  year={2024}
}
% for the older version
@article{wang2023decur,
  title={DeCUR: decoupling common \& unique representations for multimodal self-supervision},
  author={Wang, Yi and Albrecht, Conrad M and Braham, Nassim Ait Ali and Liu, Chenying and Xiong, Zhitong and Zhu, Xiao Xiang},
  journal={arXiv preprint arXiv:2309.05300},
  year={2023}
}