[Bugfix] Fix offline_inference_with_prefix.py

[tlrmchlsmth]

The example runs OOM on current main. This is the same as the fix in fix_lazy_outlines.py in #9352.

Copying @joerunde's inline comment from that PR:

I think this test was working before due to the over-estimation of peak memory usage of the model which caused a smaller KV cache to be allocated. Two LLMs both set gpu_memory_utilization=0.3, but once the first LLM uses the full 30% of the gpu, there's no space left to allocate room for the second one.

This setting is a bit confusing- how it has been coded is "The total GPU allocation may not exceed x% of the gpu memory when loading this model", but it looks like the test assumed the setting meant "You may not allocate more than x% of the gpu memory for this model, regardless of how much of the gpu memory ends up being allocated." In other words, it assumed this was a per-model limit and not a global limit on gpu memory allocation.

[github-actions]

👋 Hi! Thank you for contributing to the vLLM project.
Just a reminder: PRs would not trigger full CI run by default. Instead, it would only run fastcheck CI which starts running only a small and essential subset of CI tests to quickly catch errors. You can run other CI tests on top of those by going to your fastcheck build on Buildkite UI (linked in the PR checks section) and unblock them. If you do not have permission to unblock, ping simon-mo or khluu to add you in our Buildkite org.

Once the PR is approved and ready to go, your PR reviewer(s) can run CI to test the changes comprehensively before merging.

To run CI, PR reviewers can do one of these:

• Add ready label to the PR
• Enable auto-merge.

🚀

[DarkLight1337]

I also thought that gpu_memory_utilization is per LLM instance. This needs to be clarified in the docs IMO.

If possible, I prefer it to be per LLM instance as this would enable us to freely switch the order in which the LLM instances are created. That being said, I don't think we really support using multiple LLM instances simultaneously in the same process...

[DarkLight1337]

To unblock CI, let's merge this first. We can fix the quantization tests.and add docs in another PR.

[tlrmchlsmth]

Agreed with both comments @DarkLight1337

[comaniac]

It might be better to solve this issue in this example completely by destroying the first LLM before creating the next one. I could help fix this in another PR.