Teaching VLMs to Localize Specific Objects from In-context Examples (IPLoc)

This repository contains the code for the paper "Teaching VLMs to Localize Specific Objects from In-context Examples (IPLoc)", by Sivan Doveh et al

Environment Setup:

Prepare the Qwen2-VL environment as shown in Qwen2VL Env

Data Preparation:

1. Download and place the images in the respective folders:

   • per_seg: Download PerSeg images
   • LASOT: Download LASOT images
   • frames(PDM): Download PDM images

2. The folder structure should look like this:

   data/
   ├── ICL_tracking/
       └── video/
           ├── frames/
           ├── per_seg/
           └── LASOT/
   

Model Download:

Download our model from QWEN2-VL-ICL-LOC

Evaluation:

To evaluate the ICLoc model, use the following command:

python Loc_Qwen2VL7B.py --data_path ./data/test_data_path.json --name IPLocEval --lora_weights_path lora_pth_to_model

Test data JSON files are in the data directory, including ICL - PDM, LASOT, and PerSeg.