Weighted Ensemble Models Are Strong Continual Learners

Imad Eddine MAROUF¹  Subhankar Roy²  Enzo Tartaglione¹  Stéphane Lathuilière¹

¹Telecom-Paris, Institut Polytechnique de Paris 

²University of Trento, Italy 

The code repository for "Weighted Ensemble Models Are Strong Continual Learners" in PyTorch.

📣 Published as a conference paper at ECCV 2024

Updates

📌 Our method has been added to the PILOT library: LAMDA-PILOT.

Abstract

In this work, we study the problem of continual learning (CL) where the goal is to learn a model on a sequence of tasks, with the data from previous tasks becoming unavailable while learning on the current task data. CL is essentially a balancing act between learning new tasks (plasticity) and maintaining performance on previously learned concepts (stability). To address the stability-plasticity trade-off, we propose performing weight-ensembling of the model parameters of the previous and current task. This weight-ensembled model, which we call Continual Model Averaging (or CoMA), achieves high accuracy on the current task without deviating significantly from the previous weight configuration, ensuring stability. We also propose an improved variant of CoMA, named Continual Fisher-weighted Model Averaging (or CoFiMA), that selectively weighs each parameter in the weight ensemble by leveraging the Fisher information of the model's weights. Both variants are conceptually simple, easy to implement, and effective in achieving state-of-the-art performance on several standard CL benchmarks.

Requirement

install the conda environment using the environment.yml file

conda env create --file environement.yml

Pre-trained Models

Please download pre-trained ViT-Base models from MoCo v3 and ImageNet-21K and then put or link the pre-trained models to CoFiMA/pretrained

Dataset Preparation

For downloading and preparing the datasets used in our experiments, please refer to the instructions provided in LAMDA-PILOT. This repository contains detailed steps for setting up the standard continual learning benchmarks.

Training

to launch the training of CoMA or CoFiMA on CIFAR-100, run the following command:

python main.py --config exps/cofima/cofima_cifar.json

python main.py --config exps/cofima/cofima_imgnetr.json

Log file

The default log file for CoFiMA evaluated on the main benchmarks in Tab.1 are in CoFiMA/log.txt

Citation

If you find this work helpful, please cite our paper.

@misc{marouf2024weightedensemblemodelsstrong,
      title={Weighted Ensemble Models Are Strong Continual Learners}, 
      author={Imad Eddine Marouf and Subhankar Roy and Enzo Tartaglione and Stéphane Lathuilière},
      booktitle = {European Conference on Computer Vision (ECCV)},
      year    = {2024}
}

Acknowledgment

This repo is heavily based on PyCIL, SLCA, many thanks.