README.md

RefSAM: Efficiently Adapting Segmenting Anything Model for Referring Video Object Segmentation

Yonglin Li, Jing Zhang, Xiao Teng, Long Lan, Xinwang Liu

Abstract | Usage | Results | Statement

Introduction

This is the official repository of the paper RefSAM: Efficiently Adapting Segmenting Anything Model for Referring Video Object Segmentation

The overall pipeline of RefSAM. It mainly consists of five key components: 1) Backbone: Visual Encoder of SAM with Adapter and Text Encoder; 2) Cross-Modal MLP; 3) Hierarchical Dense Attention; 4) Mask Decoder of SAM; and 5) Implicit Tracking Module.

In this study, we present the RefSAM model, which for the first time explores the potential of SAM for RVOS by incorporating multi-view information from diverse modalities and successive frames at different timestamps. Our proposed approach adapts the original SAM model to enhance cross-modality learning by employing a lightweight Cross-Modal MLP that projects the text embedding of the referring expression into sparse and dense embeddings, serving as user-interactive prompts. Subsequently, a parameter-efficient tuning strategy is employed to effectively align and fuse the language and vision features. Through comprehensive ablation studies, we demonstrate the practical and effective design choices of our strategy. Extensive experiments conducted on Ref-Youtu-VOS and Ref-DAVIS17 datasets validate the superiority and effectiveness of our RefSAM model over existing methods.

Usage

Results

Results on RVOS datasets

Figure 2: Results on Ref-DAVIS17.

Figure 3: Results on Ref-Youtube-VOS.

A comprehensive comparison between RefSAM and existing methods.

Visualization Results

Visualization results of our RefSAM model on Ref-DAVIS17.

Figure 4: Some examples of RefSAM segmentation results on Ref-DAVIS17.

We show the visualization results of our RefSAM model. It can be seen that RefSAM is capable of effectively segmenting and tracking the referred object even in challenging scenarios, such as variations in person poses, and occlusions between instances.

Visualization of different models on Ref-DAVIS17.

Figure 5: Visualization of different models on Ref-DAVIS17. From left to right: RefSAM, ReferFormer, SAM-Track + Ground DINO, and PerSAM + Ground DINO.

Furthermore, we present the results of differnt models. It is clear that our RefSAM demonstrates significantly enhanced cross-modal understanding capability.

Model Analysis

The influence of different learning rates for the learnable modules

Figure 6: The influence of different learning rates for the learnable modules of RefSAM.

Ablation study of different module designs.

Figure 7: Ablation study of different module designs.

Ablation study of the key components

Figure 8: Ablation study of the key compoents of RefSAM

Influence of the model size of Visual Encoder

Figure 9: Influence of the model size of Visual Encoder.

Number of learnable parameters of different Models

Figure 10: Number of learnable parameters of different Models

Inference speed of different models.

Figure 11: Inference speed of different models.

Statement

This project is for research purpose only. For any other questions please contact [email protected].

Citation