[Usage]: prefix caching support for multimodal models

[mearcstapa-gqz]

Your current environment

The output of `python collect_env.py`

How would you like to use vllm

Hi I want to perform offline batch inference with prefix caching to accelerate inference.
But It looks like it is not supported yet when I initialize LLM with enable_prefix_caching=True

--enable-prefix-caching is currently not supported for multimodal models and has been disabled.
Would it be supported in near future and/or can I get some reference on how to implement it myself?
Currently I'm trying to make batch inference with prefix cache work using huggingface transformers, but it looks like I can't get padding and/or cache position and/or attention_mask right so I get gibberish outputs.

ref:
https://docs.vllm.ai/en/stable/getting_started/examples/offline_inference_with_prefix.html
https://docs.vllm.ai/en/stable/getting_started/examples/offline_inference_vision_language.html
huggingface/transformers#34232 (comment)

Before submitting a new issue...

• Make sure you already searched for relevant issues, and asked the chatbot living at the bottom right corner of the documentation page, which can answer lots of frequently asked questions.