[Misc]: VLLM docker image size

[fluidnumerics-joe]

Anything you want to discuss about vllm.

The VLLM and vllm-dev image builds for ROCm on docker hub are quite large (~22GB). I wanted to know if there is interest or are already efforts to work towards optimizing the image size to reduce download times for end users.

Before submitting a new issue...

• Make sure you already searched for relevant issues, and asked the chatbot living at the bottom right corner of the documentation page, which can answer lots of frequently asked questions.

[gshtras]

The size of these images depends on the ROCm release image sizes, vLLM adds the bare minimum on top of it.
The actual size of the nightly vllm image is 7.5GB compressed on the dockerhub.

[fluidnumerics-joe]

I see vllm-dev comes in around ~7GB, but VLLM (https://hub.docker.com/r/rocm/vllm/tags) is around 22GB.

[gshtras]

vllm-dev are the nightly images, latest and greatest, but prone to intermittent regressions.
rocm/vllm is the official release, signed off and tested and therefore its release cadence and the process itself is very different.
Sometime in the future it's likely to look more like the current nightly.

[fluidnumerics-joe]

Functionally, are both of the meant to provide the same tooling ?

[fluidnumerics-joe]

Why is there such a difference in the image sizes though ?

[gshtras]

The current ones in rocm/vllm are built on rocm/pytorch release images, which are quite large. vllm functionality is also frozen to the time of ROCm6.3 release
Nightlies use later versions of libraries such as hipblaslt, triton, torch, etc. Also, vllm is always top of tree at the time of the build there.

So, regarding

Functionally, are both of the meant to provide the same tooling ?

tl;dr, no