Evaluation of Foundation Models for Earth Observation

This repository provides tools for evaluating various foundation models on Earth Observation tasks. For detailed instructions on pretraining and deploying DOFA, please refer to the main DOFA repository.

---

Setup

To get started, ensure you have the following dependencies installed:

pip install torch==2.1.2
pip install torchvision==0.16.2
pip install numpy==1.26.4
pip install openmim
mim install mmengine
mim install "mmcv==2.1.0"
mim install mmsegmentation
pip install tensorboard
pip install loguru
pip install timm
pip install git+https://github.com/ServiceNow/geo-bench.git
pip install torchmetrics
pip install ftfy
pip install regex
pip install einops
pip install yacs
pip install kornia
pip install pydantic
pip install omegaconf
pip install wandb
pip install python-dotenv
pip install torchgeo
pip install fastparquet

Install them by:

pip install -e .

Model Weights

Pretrained model weights are available on Hugging Face. Download the necessary weights for your evaluation tasks. The SoftCon weights are available on a different HF repo. DOFA weights are available in this HF repo. SATMAE++ weights are available in this HF repo.

download the pre-trained weights and put them to the src/fm_weights folder.

---

Available Models

This repository includes the following models for evaluation:

• CROMA
• DOFA
• GFM
• RemoteCLIP
• SatMAE
• ScaleMAE
• SoftCON
• on-going...

---

Supported Datasets

The following datasets are currently supported:

• GeoBench
• Spectral Earth (in progress)

---

Adding New Models and Datasets

To add a new model or dataset for evaluation, follow these steps:

1. Add a Model Wrapper:

   • Create a new model wrapper in the foundation_models folder.
   • Add the new model to __init__.py for integration.
   • Register the model in factory.py by adding its name to make it accessible via the model_type parameter.

2. Add a Dataset Wrapper:

   • Create a new dataset wrapper in the datasets folder.
   • Register the dataset in factory.py to ensure access.

3. Configuration Setup:

   • Define the configuration settings in config.py.

---

Running the Evaluation

To run the evaluation script, use the following command:

python scripts/exp_config.py

This script initiates training for model evaluation on GeoBench with CROMA.

---

Contributing

We welcome contributions! If you'd like to add new models, datasets, or evaluation scripts, please submit a pull request, and ensure that you have tested your changes.