We use VLMEVALKIT as the evaluation tools. Please refer to QuickStart for installation.
We evaluate the models with scripts in evaluation.sh. You could modify the parameters for evaluating different benchmarks.
You should use the file run.py to replace with the original run file to evaluate the model.
eval_dataset="MMVet" # need openai api key
eval_dataset="MME MMBench_DEV_EN"
# set the llava-path to the actual path of your convllava checkpointWe would contribute the VLMEVALKIT to support our model soon.
If you want to use lmms-eval to evaluate the model. You need to first install the package:
git clone https://github.com/EvolvingLMMs-Lab/lmms-eval
cd lmms-eval
pip install -e .You should use the file eval-lmms.sh to evaluate the model. You could modify the parameters for evaluating different benchmarks.
If you are interested in RefCOCO, we provide the code in refcoco.sh.