Benchmarks测试
MTT 推理引擎性能测试
新建配置脚本perf_config.json如下：
echo \
"[
{
\"model_name\": \"$model_name\",
\"path\": \"$converted_model_dir\",
\"batchs\": [1,2,4,8],
\"prefill_token_lens\": [256,512,1024],
\"decode_token_lens\": [64]
}
]" > perf_config.json
示例文件内容:
perf_config.json
[
{
"model_name": "Qwen2.5-72B-Instruct",
"path": "/data/mtt/models_convert/Qwen2.5-72B-Instruct-converted/",
"batchs": [1,2,4,8],
"prefill_token_lens": [256,512,1024],
"decode_token_lens": [64]
}
]
然后执行以下命令开始测试：
python -m mttransformer.perf_test perf_config.json
在执行路径下，测试结果会同时打印在命令行窗口并存放在./perf_data/文件夹内的csv文件 ，输出示例如下图所示：
Qwen2.5-72B-Instruct perf finish
=========
Model Data_Type GPU_Num batch prefill_tokens decode_tokens prefill_latency single_batch_decode_tps total_decode_tps
0 Qwen2.5-72B-Instruct FP16 8 1 256 64 *** *** ***
1 Qwen2.5-72B-Instruct FP16 8 1 512 64 *** *** ***
2 Qwen2.5-72B-Instruct FP16 8 1 1024 64 *** *** ***
3 Qwen2.5-72B-Instruct FP16 8 2 256 64 *** *** ***
4 Qwen2.5-72B-Instruct FP16 8 2 512 64 *** *** ***
5 Qwen2.5-72B-Instruct FP16 8 2 1024 64 *** *** ***
6 Qwen2.5-72B-Instruct FP16 8 4 256 64 *** *** ***
7 Qwen2.5-72B-Instruct FP16 8 4 512 64 *** *** ***
8 Qwen2.5-72B-Instruct FP16 8 4 1024 64 *** *** ***
9 Qwen2.5-72B-Instruct FP16 8 8 256 64 *** *** ***
10 Qwen2.5-72B-Instruct FP16 8 8 512 64 *** *** ***
11 Qwen2.5-72B-Instruct FP16 8 8 1024 64 *** *** ***
可以通过-o指定输出路径。也可以通过-h查询完整的命令用法：
$ python -m mttransformer.perf_test -h
usage: perf_test.py [-h] [-o OUTPUT_PATH] config_path
mtt perf script
positional arguments:
config_path input config json file path
optional arguments:
-h, --help show this help message and exit
-o OUTPUT_PATH, --output-path OUTPUT_PATH
output file path
| 列名 | 含义 | 样例 |
|---|---|---|
| Model | 模型名 | Qwen2.5-72B-Instruct |
| Data_Type | 数据精度 | FP16 |
| GPU_Num | GPU数量 | 8 |
| batch | Batch大小 | 1 |
| prefill_tokens | 预填充长度 | 512 |
| decode_tokens | 解码长度 | 64 |
| prefill_latency | 预填充延迟/首字延迟/TTFT（ms） | 377 |
| single_batch_decode_tps | 单batch吞吐（token/s） | 30 |
| total_decode_tps | 总吞吐（token/s） | 60 |
vLLM-MTT 性能测试一（官网脚本）
可以使用vLLM官方提供的Benchmarks脚本进行性能测试。下文所有测试脚本均可以在/home/workspace/vllm_mtt/benchmarks路径下或者vLLM官网找到
下载数据集
wget -P /home/workspace/vllm_mtt/benchmarks https://huggingface.co/datasets/anon8231489123/ShareGPT_Vicuna_unfiltered/resolve/main/ShareGPT_V3_unfiltered_cleaned_split.json
信息
如果huggingface下载速度受限，也可以在魔搭社区下载：
git lfs install
git clone https://www.modelscope.cn/datasets/gliang1001/ShareGPT_V3_unfiltered_cleaned_split.git /home/workspace/vllm_mtt/benchmarks
离线测试
cd /home/workspace/vllm_mtt/benchmarks
python benchmark_throughput.py \
--dataset ./ShareGPT_V3_unfiltered_cleaned_split/ShareGPT_V3_unfiltered_cleaned_split.json \
--model $converted_model_dir \
--trust-remote-code \
--output-len 64 \
--kv-cache-dtype "auto" \
--device "musa" \
--tensor-parallel-size $tp_size
样例输出：
Throughput: *** requests/s, *** tokens/s
在线服务测试
首先确保按照启动OpenAI API Server中所述部署好服务。通过以下命令进行测试：
# 通过api调用模型
curl http://0.0.0.0:8000/v1/chat/completions -H "Content-Type: application/json" -d '{
"model": "'"$converted_model_dir"'",
"messages": [
{"role": "system", "content": "You are a helpful assistant."},
{"role": "user", "content": "Who won the NBA final series in 2020?"}
]
}'
启动测试：
cd /home/workspace/vllm_mtt/benchmarks
python benchmark_serving.py \
--dataset-path ./ShareGPT_V3_unfiltered_cleaned_split/ShareGPT_V3_unfiltered_cleaned_split.json \
--backend vllm \
--dataset-name sharegpt \
--model $converted_model_dir \
--trust-remote-code \
--num-prompts 1000
样例输出：
============ Serving Benchmark Result ============
Successful requests: ***
Benchmark duration (s): ***
Total input tokens: ***
Total generated tokens: ***
Request throughput (req/s): ***
Input token throughput (tok/s): ***
Output token throughput (tok/s): ***
---------------Time to First Token----------------
Mean TTFT (ms): ***
Median TTFT (ms): ***
P99 TTFT (ms): ***
-----Time per Output Token (excl. 1st token)------
Mean TPOT (ms): ***
Median TPOT (ms): ***
P99 TPOT (ms): ***
==================================================
详细参数选择可以在脚本中了解更多。
vLLM-MTT测试二（llmperf）
llmperf是一个开源的LLM性能测试项目: https://github.com/ray-project/llmperf。该项目包括性能测试和正确性测试两部分。
安装设置
git clone https://github.com/ray-project/llmperf.git
cd llmperf
pip install -e .
因llmperf依赖的tokenizer涉及使用huggingface，故需要设置HuggingFace镜像：
export HF_ENDPOINT=https://hf-mirror.com
性能测试
export OPENAI_API_KEY=openai
export OPENAI_API_BASE="http://0.0.0.0:8000/v1"
python token_benchmark_ray.py \
--model $converted_model_dir \
--mean-input-tokens 550 \
--stddev-input-tokens 150 \
--mean-output-tokens 150 \
--stddev-output-tokens 10 \
--max-num-completed-requests 100 \
--timeout 600 \
--num-concurrent-requests 5 \
--results-dir "result_outputs" \
--llm-api openai \
--additional-sampling-params '{}'
样例输出：
inter_token_latency_s
p25 = ***
p50 = ***
p75 = ***
p90 = ***
p95 = ***
p99 = ***
mean = ***
min = ***
max = ***
stddev = ***
ttft_s
p25 = ***
...
end_to_end_latency_s
p25 = ***
...
request_output_throughput_token_per_s
p25 = ***
...
number_input_tokens
p25 = ***
...
number_output_tokens
p25 = ***
...
Number Of Errored Requests: ***
Overall Output Throughput: ***
Number Of Completed Requests: ***
Completed Requests Per Minute: ***
详细报告文件存放在result_outputs文件夹下。
正确性测试
export OPENAI_API_KEY=openai
export OPENAI_API_BASE="http://0.0.0.0:8000/v1"
python llm_correctness.py \
--model $converted_model_dir \
--max-num-completed-requests 150 \
--timeout 600 \
--num-concurrent-requests 10 \
--results-dir "result_outputs"
样例输出：
Results for llm correctness test for /data/mtt/models_convert/Qwen2.5-72B-Instruct-converted/ queried with the openai api.
Errors: 0, Error rate: 0.0
Mismatched: 0, Mismatch rate: 0.0
Completed: 150
Completed without errors: 150
