推理性能

以下推理性能数据均在打开性能模式条件下测试。

Qwen3-30B-A3B-GPTQ-Int4

concurrency	input_len	output_len	TTFT (ms)	ITL (ms)	TPS(out)
1	32	32	129.14	78.17	12.53
1	128	128	414.25	79.59	12.16
1	1024	1024	546.74	81.85	12.15
1	3072	1024	1361.76	89.48	11.02
1	8192	1024	290.29	110.43	9.06

测试性能

环境准备

conda activate v1.2
pip3 install pandas==2.3.1 datasets==4.0.0 
git clone https://github.com/vllm-project/vllm.git
cd vllm
git checkout v0.7.3
cd benchmarks

运行测试

python3 benchmark_serving.py --model models/gptq-Qwen2.5-7B-Instruct/ --dataset_name random --random_input_len 128 --random_output_len 128 --num-prompts 1 --trust-remote-code --ignore-eos

备注

1. 一个窗口启动模型服务，另一个窗口运行性能测试的脚本
2. 如果报错 transformers 找不到：pip3 install transformers==4.52.4
3. 性能表格TPS(out)的值为1000/TPOT (ms)得出
4. 上述性能仅供参考,运行实际结果受具体环境,测试方法和测试数据集影响

输出示例

============ Serving Benchmark Result ============
Successful requests:                     1
Benchmark duration (s):                  13.87
Total input tokens:                      128
Total generated tokens:                  128
Request throughput (req/s):              0.07
Output token throughput (tok/s):         9.23
Total Token throughput (tok/s):          18.46
---------------Time to First Token----------------
Mean TTFT (ms):                          620.98
Median TTFT (ms):                        620.98
P99 TTFT (ms):                           620.98
-----Time per Output Token (excl. 1st token)------
Mean TPOT (ms):                          104.31
Median TPOT (ms):                        104.31
P99 TPOT (ms):                           104.31
---------------Inter-token Latency----------------
Mean ITL (ms):                           104.31
Median ITL (ms):                         103.01
P99 ITL (ms):                            125.80
==================================================