[Usage]: How to make model response information appear in the vllm backend logs

[nora647]

Your current environment

I am deploying a qwen2.5-7b model via vllm serve Qwen/Qwen2.5-7B-Instruct with api calls using the langchain framework.

vllm serve Qwen/Qwen2.5-7B-Instruct

But in the process of using it, I found that the vllm background log only records the log of requests to llm, and does not record the log of llm replies, which is very inconvenient in program debugging.

I would like to know how to make vllm background log can record model reply information in real time.For example, as shown in the following figure

How would you like to use vllm

I would like to know how to make vllm background log can record model reply information in real time.For example, as shown in the following figure.Thank you!

Before submitting a new issue...

• Make sure you already searched for relevant issues, and asked the chatbot living at the bottom right corner of the documentation page, which can answer lots of frequently asked questions.

[DarkLight1337]

The output isn't logged in our code, you'll have to edit chat_completion_full_generator and/or chat_completion_stream_generator inside vllm/entrypoints/openai/serving_chat.py to log this information.