[Badcase]: 性能评测报告A100卡只有1并发报告；8并发，16并发请求性能数据能达到多少？

[AuSong]

Model Series

Qwen2.5

What are the models used?

Qwen2.5-7B-Instruct

What is the scenario where the problem happened?

Qwen2.5-7B-Instruct 性能

Is this badcase known and can it be solved using avaiable techniques?

• I have followed the GitHub README.
• I have checked the Qwen documentation and cannot find a solution there.
• I have checked the documentation of the related framework and cannot find useful information.
• I have searched the issues and there is not a similar one.

Information about environment

NVIDIA A100 80GB

CUDA 12.1

vLLM 0.6.3

Pytorch 2.4.0

Flash Attention 2.6.3

Transformers 4.46.0

Description

Steps to reproduce

This happens to Qwen2.5-xB-Instruct-xxx and xxx.
The badcase can be reproduced with the following steps:

1. ...
2. ...

The following example input & output can be used:

system: ...
user: ...
...

Expected results

The results are expected to be ...

Attempts to fix

I have tried several ways to fix this, including:

1. adjusting the sampling parameters, but ...
2. prompt engineering, but ...

Anything else helpful for investigation

I find that this problem also happens to ...

[jklj077]

I believe the report provided performance results for a single request, which is not conceptually the same with concurrency one.

With concurrent requests, the maximum throughput (tokens per second) should be stable for vllm whatever the concurrency. However, the latency (e.g., time to first token) for each request varies significantly depending on the vllm configuration and the data distribution.

We recommend benchmarking the results on your own with the vllm official script at https://github.com/vllm-project/vllm/blob/main/benchmarks/benchmark_serving.py which can simulate max concurrency, request rate, burstiness, etc., with a sample dataset.