VLLM takes alot of time to start, can we use the docker image?

[Stealthwriter]

Hi,

I'm trying to use sky serve with the docker image of vllm, and my cloud provider is runpod, which YAML should I use?

[chris-aeviator]

you should be good with picking a vllm container that suites your needs and load it as described in https://skypilot.readthedocs.io/en/latest/examples/docker-containers.html#docker-containers-as-runtime-environments

[romilbhardwaj]

We have an example using vllm docker image here: https://github.com/skypilot-org/skypilot/blob/master/llm/vllm/serve-openai-api-docker.yaml

Note that runpod has some limitations and this yaml may not work.