djl serving + vllm deploy sagemaker endpoint miss token metric

[DongZhaoXiong]

Description

TOKENIZER=/home/ec2-user/workspace/MyProject/aitao_llm/llm_container/tokenizer.json awscurl -c 1 -N 2 -X POST -n sagemaker https://runtime.sagemaker.cn-northwest-1.amazonaws.com.cn/endpoints/qwen25-32b-model-2024-12-21-09-03-10-213-endpoint/invocations -d '{"inputs": "hello", "parameters":{"temperature": 0.7, "top_p": 0.8, "max_tokens":256}}' -H 'Content-Type: application/json' -t --profile GA --region cn-northwest-1
Total time: 2511.73 ms.
Non 200 responses: 0, error rate: 0.00
Concurrent clients: 1
Total requests: 2
TPS: 0.86/s
Average Token Latency: Infinity ms.
Average Latency: 1166.49 ms.
P50: 1715.55 ms.
P90: 1715.55 ms.
P99: 1715.55 ms.
Time to first byte: 867.82 ms.
TTFB_P50: 1118.42 ms.
TTFB_P90: 1118.42 ms.
TTFB_P99: 1118.42 ms