推理性能

模型性能数据

以下数据来自 SGLang-MUSA 0.5.18+20261010 在 M1000 AImodule 端侧环境的实测结果。均在性能模式 + compute_only 环境下测试，数值供参考。

测试环境

• 端设备：M1000 AImodule，64G 统一内存
• 系统版本：V1.5.0
• MUSA 版本：5.2.1
• 服务参数：max-total-tokens=16384
• 文本测试并发：1
• 文本测试输入 / 输出：无前缀场景为 128/100、250/100、500/100、1000/100、2000/100；带 prefix 场景为 500/100、1000/100、2000/100、4000/100、8000/100、16000/100，其中 PrefillLen=250

测试说明

测试指标说明如下：

• Prefill(tps)：Prefill 阶段吞吐，单位 token/s。
• TTFT(ms)：首 token 延迟，单位 ms。
• Decode(tps)：Decode 阶段吞吐，单位 token/s。
• TPOT(ms)：单个输出 token 平均耗时，单位 ms。
• Profile：测试 profile，格式为 prefill_len:input_len。
• Vision(fps)：视觉编码帧率，单位 frame/s。
• Precision：模型量化精度。

性能复现流程

解压得到安装包目录：

tar -zxvf 20260930_M1000_1.5.0_sglang.tar.gz
cd 20260930_M1000_1.5.0_sglang

启动服务前先预编译 TileLang kernel；如果你已经在 快速开始 里执行过这一步，这里可以直接跳过：

./scripts/ci/musa/tilelang_precompile/precompile_tilelang.sh --model qwen35,qwen3,qwen35_fq,qwen36_fq,qwen35_dflash,qwen35_fq_dflash,qwen3vl

分别启动服务并运行 benchmark。每组测试使用两个终端：启动 端运行服务，运行 benchmark 端执行对应测试。benchmark 通过 MODEL_KEY 选择对应模型配置。

启动方式和参考性能数据

Qwen3-30B-A3B-GPTQ-Int4 (MoE GPTQ，GPTQ INT4 (W4A16))

启动：

scripts/ci/musa/start_qwen3_server.sh

运行 benchmark：

MODEL_KEY=qwen3 scripts/ci/musa/run_tilelang_benchmark.sh

Model	Size	PrefillLen(tokens)	Ctx(K)	Batch	NChip	Input(tokens)	Output(tokens)	Prefill(tps)	TTFT(ms)	Decode(tps)	TPOT(ms)	Profile	Prefix(tokens)	MeasuredOutput(tokens)
Qwen3	30b_a3b	128	16	1	1	128	16	217.136	589.493	24.989	40.018	128:128	0	16
Qwen3	30b_a3b	250	16	1	1	250	16	278.068	899.06	24.86	40.226	250:250	0	16
Qwen3	30b_a3b	500	16	1	1	500	16	233.452	2141.764	24.49	40.833	500:500	0	16
Qwen3	30b_a3b	1000	16	1	1	1000	16	280.011	3571.292	24.041	41.596	1000:1000	0	16
Qwen3	30b_a3b	2000	16	1	1	2000	16	293.807	6807.185	23.218	43.07	2000:2000	0	16
Qwen3	30b_a3b	250	16	1	1	500	16	195.707	1277.423	26.119	38.287	250:500	250	16
Qwen3	30b_a3b	250	16	1	1	1000	16	193.575	1291.489	24.061	41.561	250:1000	750	16
Qwen3	30b_a3b	250	16	1	1	2000	16	164.335	1521.287	23.166	43.166	250:2000	1750	16
Qwen3	30b_a3b	250	16	1	1	4000	16	148.625	1682.083	21.823	45.823	250:4000	3750	16
Qwen3	30b_a3b	250	16	1	1	8000	16	102.574	2437.266	19.497	51.289	250:8000	7750	16
Qwen3	30b_a3b	250	16	1	1	16000	16	61.944	4035.902	16.234	61.599	250:16000	15750	16

Qwen3.5-35B-A3B-GPTQ-Int4 (MoE GPTQ，GPTQ INT4 (W4A16))

启动：

scripts/ci/musa/start_qwen35_server.sh

运行 benchmark：

MODEL_KEY=qwen35 scripts/ci/musa/run_tilelang_benchmark.sh

Model	Size	PrefillLen(tokens)	Ctx(K)	Batch	NChip	Input(tokens)	Output(tokens)	Prefill(tps)	TTFT(ms)	Decode(tps)	TPOT(ms)	Profile	Prefix(tokens)	MeasuredOutput(tokens)
Qwen3.5	35b_a3b	128	16	1	1	128	16	245.902	520.533	15.517	64.445	128:128	0	16
Qwen3.5	35b_a3b	250	16	1	1	250	16	272.974	915.838	15.464	64.668	250:250	0	16
Qwen3.5	35b_a3b	500	16	1	1	500	16	228.566	2187.553	15.425	64.828	500:500	0	16
Qwen3.5	35b_a3b	1000	16	1	1	1000	16	273.103	3661.622	15.399	64.939	1000:1000	0	16
Qwen3.5	35b_a3b	2000	16	1	1	2000	16	307.999	6493.522	15.321	65.271	2000:2000	0	16
Qwen3.5	35b_a3b	250	16	1	1	500	16	183.388	1363.231	15.425	64.829	250:500	250	16
Qwen3.5	35b_a3b	250	16	1	1	1000	16	189.364	1320.21	15.376	65.035	250:1000	750	16
Qwen3.5	35b_a3b	250	16	1	1	2000	16	185.4	1348.439	15.332	65.225	250:2000	1750	16
Qwen3.5	35b_a3b	250	16	1	1	4000	16	160.624	1556.434	15.15	66.008	250:4000	3750	16
Qwen3.5	35b_a3b	250	16	1	1	8000	16	162.628	1537.252	14.809	67.528	250:8000	7750	16
Qwen3.5	35b_a3b	250	16	1	1	16000	16	118.907	2102.483	14.236	70.243	250:16000	15750	16

Qwen3.5-35B-A3B-GPTQ-Int4 (MoE GPTQ，GPTQ INT4 (W4A16)) 开启 MTP 投机解码

启动：

QWEN35_EXTRA_SERVER_ARGS="--mem-fraction-static 0.97 --speculative-algorithm NEXTN --speculative-num-steps 3 --speculative-eagle-topk 1 --speculative-num-draft-tokens 4" scripts/ci/musa/start_qwen35_server.sh

运行 benchmark (GSM8K测试集):

MODEL_KEY=qwen35_mtp scripts/ci/musa/run_tilelang_benchmark.sh

Model	Size	PrefillLen(tokens)	Batch	NChip	Input(tokens)	Output(tokens)	Prefill(tps)	TTFT(ms)	Decode(tps)	TPOT(ms)	Prefix(tokens)	Prompt Index	accept rate((acclen-1)/(4-1))	Prediction	Ground Truth	Correct
Qwen3.5	35b_a3b	869	1	1	869	156	217.32	3998.68	21.64	46.21	0	6	0.7569	64	64	TRUE
Qwen3.5	35b_a3b	855	1	1	855	116	247.41	3455.85	24.11	41.48	0	7	0.8854	260	260	TRUE
Qwen3.5	35b_a3b	884	1	1	884	254	250.53	3528.45	21.96	45.53	0	8	0.7749	160	160	TRUE
Qwen3.5	35b_a3b	919	1	1	919	259	255.23	3600.62	23.2	43.1	0	9	0.8423	45	45	TRUE
Qwen3.5	35b_a3b	875	1	1	875	154	249.16	3511.76	23.79	42.03	0	10	0.8682	460	460	TRUE
Qwen3.5	35b_a3b	875	1	1	875	144	249.28	3510.1	23.92	41.8	0	11	0.875	366	366	TRUE
Qwen3.5	35b_a3b	879	1	1	879	182	250.28	3512.06	24.22	41.29	0	12	0.8733	694	694	TRUE
Qwen3.5	35b_a3b	882	1	1	882	179	250.43	3521.98	21.66	46.17	0	13	0.7636	12	13	FALSE
Qwen3.5	35b_a3b	872	1	1	872	373	247.4	3524.71	24.28	41.18	0	14	0.8824	18	18	TRUE
Qwen3.5	35b_a3b	867	1	1	867	222	248.79	3484.93	23.82	41.98	0	15	0.871	60	60	TRUE
Qwen3.5	35b_a3b	909	1	1	909	123	253.07	3591.88	24	41.67	0	16	0.8824	125	125	TRUE
Qwen3.5	35b_a3b	868	1	1	868	61	247.87	3501.9	22.4	44.64	0	17	0.7778	230	230	TRUE
Qwen3.5	35b_a3b	872	1	1	872	148	248.74	3505.7	24.63	40.6	0	18	0.9	57500	57500	TRUE
Qwen3.5	35b_a3b	843	1	1	843	80	245.34	3436.08	21.21	47.14	0	19	0.7467	7	7	TRUE
Qwen3.5	35b_a3b	880	1	1	880	178	250.39	3514.49	22.34	44.76	0	20	0.7799	6	6	TRUE

Qwen3.5-35B-A3B-GPTQ-Int4 (MoE GPTQ，GPTQ INT4 (W4A16)) 开启 DFLASH 投机解码

启动：

scripts/ci/musa/start_qwen35_dflash_server.sh

运行 benchmark (GSM8K测试集):

MODEL_KEY=qwen35_dflash scripts/ci/musa/run_tilelang_benchmark.sh

Model	Size	PrefillLen(tokens)	Batch	NChip	Input(tokens)	Output(tokens)	Prefill(tps)	TTFT(ms)	Decode(tps)	TPOT(ms)	Prefix(tokens)	Prediction	Ground Truth	accept len	Correct
Qwen3.5-35B-A3B-GPTQ-Int4	35B-A3B	54	1		54	172	90.89	594.11	28.16	35.51	0	3	3	6.37	TRUE
Qwen3.5-35B-A3B-GPTQ-Int4	35B-A3B	85	1		85	415	115.93	733.19	30.85	32.42	0	70000	70000	6.917	TRUE
Qwen3.5-35B-A3B-GPTQ-Int4	35B-A3B	63	1		63	253	102.57	614.19	31.85	31.4	0	540	540	7.229	TRUE
Qwen3.5-35B-A3B-GPTQ-Int4	35B-A3B	139	1		139	439	144.28	963.38	30.53	32.75	0	20	20	6.859	TRUE
Qwen3.5-35B-A3B-GPTQ-Int4	35B-A3B	83	1		83	493	112.83	735.62	26.36	37.94	0	64	64	5.869	TRUE
Qwen3.5-35B-A3B-GPTQ-Int4	35B-A3B	69	1		69	323	100.04	689.69	43.92	22.77	0	260	260	10.094	TRUE
Qwen3.5-35B-A3B-GPTQ-Int4	35B-A3B	98	1		98	368	127.6	768.02	24.17	41.37	0	160	160	5.412	TRUE
Qwen3.5-35B-A3B-GPTQ-Int4	35B-A3B	132	1		132	600	139.6	945.55	32.56	30.72	0	45	45	7.317	TRUE
Qwen3.5-35B-A3B-GPTQ-Int4	35B-A3B	88	1		88	334	118.58	742.09	28.53	35.05	0	460	460	6.423	TRUE
Qwen3.5-35B-A3B-GPTQ-Int4	35B-A3B	88	1		88	435	116.97	752.34	35.88	27.87	0	366	366	8.208	TRUE
Qwen3.5-35B-A3B-GPTQ-Int4	35B-A3B	93	1		93	445	122.73	757.75	42.93	23.29	0	694	694	9.889	TRUE
Qwen3.5-35B-A3B-GPTQ-Int4	35B-A3B	95	1		95	622	127.28	746.37	23.14	43.21	0	12	13	5.14	FALSE
Qwen3.5-35B-A3B-GPTQ-Int4	35B-A3B	86	1		86	616	115.32	745.77	33.42	29.92	0	18	18	7.605	TRUE
Qwen3.5-35B-A3B-GPTQ-Int4	35B-A3B	80	1		80	508	111.01	720.66	33.8	29.58	0	60	60	7.697	TRUE
Qwen3.5-35B-A3B-GPTQ-Int4	35B-A3B	122	1		122	349	146.22	834.38	34.95	28.61	0	125	125	7.932	TRUE
Qwen3.5-35B-A3B-GPTQ-Int4	35B-A3B	81	1		81	343	112.32	721.18	23.98	41.7	0	230	230	5.359	TRUE
Qwen3.5-35B-A3B-GPTQ-Int4	35B-A3B	86	1		86	472	117.13	734.22	31.11	32.14	0	57500	57500	7.045	TRUE
Qwen3.5-35B-A3B-GPTQ-Int4	35B-A3B	57	1		57	279	94.61	602.46	30.21	33.1	0	7	7	6.805	TRUE
Qwen3.5-35B-A3B-GPTQ-Int4	35B-A3B	93	1		93	398	124.06	749.65	30.79	32.48	0	6	6	6.982	TRUE
Qwen3.5-35B-A3B-GPTQ-Int4	35B-A3B	91	1		91	444	121.26	750.46	33.77	29.61	0	15	15	7.655	TRUE
Qwen3.5-35B-A3B-GPTQ-Int4	35B-A3B	74	1		74	279	106.01	698.08	24.89	40.17	0	14	14	5.58	TRUE
Qwen3.5-35B-A3B-GPTQ-Int4	35B-A3B	83	1		83	221	113.78	729.46	40.62	24.62	0	7	7	9.208	TRUE
Qwen3.5-35B-A3B-GPTQ-Int4	35B-A3B	68	1		68	216	99.78	681.52	30.05	33.28	0	8	8	6.75	TRUE
Qwen3.5-35B-A3B-GPTQ-Int4	35B-A3B	67	1		67	591	97.44	687.59	33.52	29.83	0	26	26	7.577	TRUE
Qwen3.5-35B-A3B-GPTQ-Int4	35B-A3B	95	1		95	405	126.33	752	35.05	28.53	0	2	2	7.941	TRUE
Qwen3.5-35B-A3B-GPTQ-Int4	35B-A3B	97	1		97	450	127.43	761.18	35.67	28.03	0	243	243	8.182	TRUE
Qwen3.5-35B-A3B-GPTQ-Int4	35B-A3B	85	1		85	334	117.51	723.35	32.97	30.33	0	16	16	7.422	TRUE
Qwen3.5-35B-A3B-GPTQ-Int4	35B-A3B	78	1		78	263	109.43	712.77	28.34	35.28	0	25	25	6.415	TRUE
Qwen3.5-35B-A3B-GPTQ-Int4	35B-A3B	97	1		97	337	124.22	780.86	30.09	33.23	0	104	104	6.878	TRUE
Qwen3.5-35B-A3B-GPTQ-Int4	35B-A3B	67	1		67	340	96.81	692.1	33.75	29.63	0	109	109	7.727	TRUE
Qwen3.5-35B-A3B-GPTQ-Int4	35B-A3B	91	1		91	373	119.44	761.9	31.93	31.32	0	80	80	7.314	TRUE
Qwen3.5-35B-A3B-GPTQ-Int4	35B-A3B	69	1		69	231	101.07	682.72	29.09	34.38	0	35	35	6.6	TRUE

Qwen3.5_FQ-35B-A3B (MoE GPTQ，GPTQ INT4 (W4A16))

启动：

scripts/ci/musa/start_qwen35_fq_server.sh

运行 benchmark：

MODEL_KEY=qwen35_fq scripts/ci/musa/run_tilelang_benchmark.sh

Model	Size	PrefillLen(tokens)	Ctx(K)	Batch	NChip	Input(tokens)	Output(tokens)	Prefill(tps)	TTFT(ms)	Decode(tps)	TPOT(ms)	Profile	Prefix(tokens)	MeasuredOutput(tokens)
Qwen3.5_FQ	35b_a3b	128	16	1	1	128	16	182.513	701.322	22.57	44.306	128:128	0	16
Qwen3.5_FQ	35b_a3b	250	16	1	1	250	16	221.779	1127.249	22.505	44.434	250:250	0	16
Qwen3.5_FQ	35b_a3b	500	16	1	1	500	16	260.048	1922.724	22.416	44.611	500:500	0	16
Qwen3.5_FQ	35b_a3b	1000	16	1	1	1000	16	290.103	3447.054	22.339	44.766	1000:1000	0	16
Qwen3.5_FQ	35b_a3b	2000	16	1	1	2000	16	285.369	7008.463	22.197	45.05	2000:2000	0	16
Qwen3.5_FQ	35b_a3b	250	16	1	1	500	16	124.342	2010.588	22.411	44.62	250:500	250	16
Qwen3.5_FQ	35b_a3b	250	16	1	1	1000	16	128.515	1945.298	22.353	44.736	250:1000	750	16
Qwen3.5_FQ	35b_a3b	250	16	1	1	2000	16	125.136	1997.823	22.207	45.031	250:2000	1750	16
Qwen3.5_FQ	35b_a3b	250	16	1	1	4000	16	114.53	2182.827	21.859	45.747	250:4000	3750	16
Qwen3.5_FQ	35b_a3b	250	16	1	1	8000	16	147.415	1695.887	21.143	47.298	250:8000	7750	16
Qwen3.5_FQ	35b_a3b	250	16	1	1	16000	16	110.979	2252.677	19.885	50.288	250:16000	15750	16

Qwen3.5_FQ-35B-A3B (MoE GPTQ，GPTQ INT4 (W4A16)) 开启 MTP 投机解码

启动：

QWEN35_FQ_EXTRA_SERVER_ARGS="--speculative-algorithm NEXTN --speculative-num-steps 3 --speculative-eagle-topk 1 --speculative-num-draft-tokens 4" scripts/ci/musa/start_qwen35_fq_server.sh

运行 benchmark (GSM8K测试集):

MODEL_KEY=qwen35_mtp PORT=30139 scripts/ci/musa/run_tilelang_benchmark.sh

Model	Size	PrefillLen(tokens)	Batch	NChip	Input(tokens)	Output(tokens)	Prefill(tps)	TTFT(ms)	Decode(tps)	TPOT(ms)	Prompt Index	accept rate((acclen-1)/(4-1))	accept rate(acclen/4)	Prediction	Ground Truth	Correct
Qwen3.5	35b_a3b	869	1	1	869	340	234.97	3698.29	24.32	41.12	6	0.713	0.787	64	64	TRUE
Qwen3.5	35b_a3b	855	1	1	855	111	232.83	3672.25	28.55	35.03	7	0.9	0.925	260	260	TRUE
Qwen3.5	35b_a3b	884	1	1	884	196	239.34	3693.52	24.82	40.29	8	0.7432	0.8033	160	160	TRUE
Qwen3.5	35b_a3b	919	1	1	919	264	232.68	3949.66	27.14	36.85	9	0.84	0.88	45	45	TRUE
Qwen3.5	35b_a3b	875	1	1	875	147	237.22	3688.56	25.21	39.67	10	0.763	0.8167	460	460	TRUE
Qwen3.5	35b_a3b	875	1	1	875	160	236.71	3696.54	26.81	37.3	11	0.8261	0.8696	366	366	TRUE
Qwen3.5	35b_a3b	879	1	1	879	182	237.64	3698.87	27.53	36.33	12	0.8497	0.8922	694	694	TRUE
Qwen3.5	35b_a3b	882	1	1	882	169	239.77	3678.48	23.61	42.36	13	0.703	0.7682	12	13	FALSE
Qwen3.5	35b_a3b	872	1	1	872	386	235.08	3709.35	27.03	37	14	0.8424	0.8773	18	18	TRUE
Qwen3.5	35b_a3b	867	1	1	867	181	234.98	3689.71	25.86	38.67	15	0.7901	0.838	60	60	TRUE
Qwen3.5	35b_a3b	909	1	1	909	157	231.16	3932.27	25.72	38.88	16	0.7872	0.8351	125	125	TRUE
Qwen3.5	35b_a3b	868	1	1	868	68	234.84	3696.15	27.43	36.46	17	0.8596	0.8947	230	230	TRUE
Qwen3.5	35b_a3b	872	1	1	872	151	236.71	3683.84	27.76	36.03	18	0.881	0.8988	57500	57500	TRUE
Qwen3.5	35b_a3b	843	1	1	843	78	229.49	3673.4	23.99	41.69	19	0.7067	0.78	7	7	TRUE
Qwen3.5	35b_a3b	880	1	1	880	179	238.32	3692.55	24.25	41.24	20	0.7076	0.7851	6	6	TRUE

Qwen3.5_FQ-35B-A3B(MoE GPTQ，GPTQ INT4 (W4A16)) 开启 DFLASH 投机解码

启动：

scripts/ci/musa/start_qwen35_fq_dflash_server.sh

运行 benchmark (GSM8K测试集):

MODEL_KEY=qwen35_fq_dflash scripts/ci/musa/run_tilelang_benchmark.sh

Model	Size	PrefillLen(tokens)	Batch	NChip	Input(tokens)	Output(tokens)	Prefill(tps)	TTFT(ms)	Decode(tps)	TPOT(ms)	Prefix(tokens)	Prediction	Ground Truth	accept len	Correct
Qwen3.5_FQ+dflash	35B-A3B	54	1		54	166	99.19	544.4	21.99	45.47	0	3	3	4.882	TRUE
Qwen3.5_FQ+dflash	35B-A3B	85	1		85	411	122.74	692.5	31.41	31.84	0	70000	70000	6.85	TRUE
Qwen3.5_FQ+dflash	35B-A3B	63	1		63	302	111.23	566.42	31.11	32.15	0	540	540	6.864	TRUE
Qwen3.5_FQ+dflash	35B-A3B	139	1		139	455	150.68	922.51	33.6	29.76	0	20	20	7.339	TRUE
Qwen3.5_FQ+dflash	35B-A3B	83	1		83	508	120.07	691.25	31.53	31.72	0	64	64	6.865	TRUE
Qwen3.5_FQ+dflash	35B-A3B	69	1		69	332	106.53	647.68	39.92	25.05	0	260	260	8.973	TRUE
Qwen3.5_FQ+dflash	35B-A3B	98	1		98	438	135.22	724.76	27.08	36.93	0	160	160	5.919	TRUE
Qwen3.5_FQ+dflash	35B-A3B	132	1		132	619	146.95	898.26	30.13	33.18	0	45	45	6.585	TRUE
Qwen3.5_FQ+dflash	35B-A3B	88	1		88	323	125.78	699.66	29.39	34.03	0	460	460	6.46	TRUE
Qwen3.5_FQ+dflash	35B-A3B	88	1		88	413	125.18	703	40.01	25	0	366	366	8.978	TRUE
Qwen3.5_FQ+dflash	35B-A3B	93	1		93	504	129.94	715.72	46.41	21.55	0	694	694	10.5	TRUE
Qwen3.5_FQ+dflash	35B-A3B	95	1		95	1236	133.2	713.21	17.59	56.84	0	12	13	3.768	FALSE
Qwen3.5_FQ+dflash	35B-A3B	86	1		86	613	122.1	704.35	31.35	31.9	0	18	18	6.888	TRUE
Qwen3.5_FQ+dflash	35B-A3B	80	1		80	565	117.64	680.06	36.12	27.68	0	60	60	7.958	TRUE
Qwen3.5_FQ+dflash	35B-A3B	122	1		122	358	153	797.41	31.51	31.74	0	125	125	7.02	TRUE
Qwen3.5_FQ+dflash	35B-A3B	81	1		81	296	119.2	679.51	24.99	40.02	0	230	230	5.481	TRUE
Qwen3.5_FQ+dflash	35B-A3B	86	1		86	1080	123.93	693.97	22.88	43.71	0	57500	57500	4.977	TRUE
Qwen3.5_FQ+dflash	35B-A3B	57	1		57	281	102.45	556.35	32.53	30.74	0	7	7	7.205	TRUE
Qwen3.5_FQ+dflash	35B-A3B	93	1		93	437	130.46	712.88	32.68	30.6	0	6	6	7.283	TRUE
Qwen3.5_FQ+dflash	35B-A3B	91	1		91	484	127.72	712.49	32.2	31.05	0	15	15	7.118	TRUE
Qwen3.5_FQ+dflash	35B-A3B	74	1		74	374	112.71	656.55	21.21	47.14	0	14	14	4.617	TRUE
Qwen3.5_FQ+dflash	35B-A3B	83	1		83	219	120.76	687.32	38.53	25.95	0	7	7	8.76	TRUE
Qwen3.5_FQ+dflash	35B-A3B	68	1		68	232	105.77	642.9	34.68	28.83	0	8	8	7.733	TRUE
Qwen3.5_FQ+dflash	35B-A3B	67	1		67	498	104.05	643.89	34.13	29.3	0	26	26	7.545	TRUE
Qwen3.5_FQ+dflash	35B-A3B	95	1		95	305	134.06	708.62	28.65	34.91	0	2	2	6.354	TRUE
Qwen3.5_FQ+dflash	35B-A3B	97	1		97	688	134.03	723.7	42.45	23.56	0	243	243	9.556	TRUE
Qwen3.5_FQ+dflash	35B-A3B	85	1		85	275	123.38	688.92	27.64	36.17	0	16	16	6.111	TRUE
Qwen3.5_FQ+dflash	35B-A3B	78	1		78	328	116.18	671.37	34.2	29.24	0	25	25	7.628	TRUE
Qwen3.5_FQ+dflash	35B-A3B	97	1		97	321	131.44	737.96	34.53	28.96	0	104	104	7.829	TRUE
Qwen3.5_FQ+dflash	35B-A3B	67	1		67	297	104.04	643.97	35.35	28.29	0	109	109	8.027	TRUE
Qwen3.5_FQ+dflash	35B-A3B	91	1		91	305	126.36	720.14	35.65	28.05	0	80	80	8.026	TRUE
Qwen3.5_FQ+dflash	35B-A3B	69	1		69	322	107.39	642.52	30.85	32.41	0	35	35	6.851	TRUE

Qwen3.5-35B-A3B-GPTQ-Int4 (MoE GPTQ, GPTQ INT4 (W4A16)) 开启 INT8 KV 量化

启动:

scripts/ci/musa/start_qwen35_server.sh --kv-cache-dtype int8

运行 benchmark:

MODEL_KEY=qwen35 scripts/ci/musa/run_tilelang_benchmark.sh

Model	Size	PrefillLen(tokens)	Ctx(K)	Batch	NChip	Input(tokens)	Output(tokens)	Prefill(tps)	TTFT(ms)	Decode(tps)	TPOT(ms)	Profile	Prefix(tokens)
Qwen3.5	35b_a3b	128	16	1	1	128	16	240.492	532.242	15.688	63.742	128:128	0
Qwen3.5	35b_a3b	250	16	1	1	250	16	268.559	930.893	15.618	64.031	250:250	0
Qwen3.5	35b_a3b	500	16	1	1	500	16	202.666	2467.119	15.512	64.467	500:500	0
Qwen3.5	35b_a3b	1000	16	1	1	1000	16	257.085	3889.769	15.545	64.331	1000:1000	0
Qwen3.5	35b_a3b	2000	16	1	1	2000	16	308.777	6477.163	15.556	64.285	2000:2000	0
Qwen3.5	35b_a3b	250	16	1	1	500	16	183.687	1361.014	15.516	64.451	250:500	250
Qwen3.5	35b_a3b	250	16	1	1	1000	16	188.399	1326.97	15.532	64.381	250:1000	750
Qwen3.5	35b_a3b	250	16	1	1	2000	16	184.649	1353.919	15.554	64.292	250:2000	1750
Qwen3.5	35b_a3b	250	16	1	1	4000	16	135.887	1839.759	15.417	64.864	250:4000	3750
Qwen3.5	35b_a3b	250	16	1	1	8000	16	140.493	1779.453	15.266	65.507	250:8000	7750
Qwen3.5	35b_a3b	250	16	1	1	16000	16	107.242	2331.185	14.953	66.874	250:16000	15750

Qwen3.6_FQ-35B-A3B (MoE GPTQ，GPTQ INT4 (W4A16))

启动：

scripts/ci/musa/start_qwen36_fq_server.sh

运行 benchmark：

MODEL_KEY=qwen36_fq scripts/ci/musa/run_tilelang_benchmark.sh

Model	Size	PrefillLen(tokens)	Ctx(K)	Batch	NChip	Input(tokens)	Output(tokens)	Prefill(tps)	TTFT(ms)	Decode(tps)	TPOT(ms)	Profile	Prefix(tokens)	MeasuredOutput(tokens)
Qwen3.6_FQ	35b_a3b	128	16	1	1	128	16	183.526	697.449	22.72	44.014	128:128	0	16
Qwen3.6_FQ	35b_a3b	250	16	1	1	250	16	231.367	1080.532	22.623	44.203	250:250	0	16
Qwen3.6_FQ	35b_a3b	500	16	1	1	500	16	209.303	2388.887	22.545	44.355	500:500	0	16
Qwen3.6_FQ	35b_a3b	1000	16	1	1	1000	16	296.768	3369.639	22.494	44.456	1000:1000	0	16
Qwen3.6_FQ	35b_a3b	2000	16	1	1	2000	16	291.555	6859.761	22.362	44.719	2000:2000	0	16
Qwen3.6_FQ	35b_a3b	250	16	1	1	500	16	164.489	1519.855	22.56	44.327	250:500	250	16
Qwen3.6_FQ	35b_a3b	250	16	1	1	1000	16	170.269	1468.261	22.479	44.485	250:1000	750	16
Qwen3.6_FQ	35b_a3b	250	16	1	1	2000	16	165.775	1508.068	22.326	44.79	250:2000	1750	16
Qwen3.6_FQ	35b_a3b	250	16	1	1	4000	16	147.109	1699.419	21.997	45.461	250:4000	3750	16
Qwen3.6_FQ	35b_a3b	250	16	1	1	8000	16	149.309	1674.378	21.289	46.972	250:8000	7750	16
Qwen3.6_FQ	35b_a3b	250	16	1	1	16000	16	112.29	2226.383	20.046	49.886	250:16000	15750	16

Qwen3.6_FQ-35B-A3B(MoE GPTQ，GPTQ INT4 (W4A16)) 开启 DFLASH 投机解码

启动：

scripts/ci/musa/start_qwen36_fq_dflash_server.sh

运行 benchmark (GSM8K测试集):

MODEL_KEY=qwen36_fq_dflash scripts/ci/musa/run_tilelang_benchmark.sh

Model	Size	PrefillLen(tokens)	Batch	NChip	Input(tokens)	Output(tokens)	Prefill(tps)	TTFT(ms)	Decode(tps)	TPOT(ms)	Prefix(tokens)	Prompt Index	Prediction	Ground Truth	accept len	Correct
Qwen3.6_FQ+dflash	35B-A3B	54	1		54	245	99.19	544.42	29.49	33.91	0	1	3	3	6.622	TRUE
Qwen3.6_FQ+dflash	35B-A3B	85	1		85	562	123.08	690.62	26.93	37.13	0	2	70000	70000	5.854	TRUE
Qwen3.6_FQ+dflash	35B-A3B	63	1		63	227	110.99	567.61	28.35	35.27	0	3	540	540	6.306	TRUE
Qwen3.6_FQ+dflash	35B-A3B	139	1		139	322	152.72	910.14	29.7	33.67	0	4	20	20	6.571	TRUE
Qwen3.6_FQ+dflash	35B-A3B	83	1		83	537	120.74	687.45	28.44	35.16	0	5	64	64	6.172	TRUE
Qwen3.6_FQ+dflash	35B-A3B	69	1		69	406	106.53	647.71	44.4	22.52	0	6	260	260	10.15	TRUE
Qwen3.6_FQ+dflash	35B-A3B	98	1		98	548	133.68	733.11	27.74	36.05	0	7	160	160	6.089	TRUE
Qwen3.6_FQ+dflash	35B-A3B	132	1		132	660	148.71	887.65	34.26	29.19	0	8	45	45	7.5	TRUE
Qwen3.6_FQ+dflash	35B-A3B	88	1		88	394	126.49	695.73	27.39	36.51	0	9	460	460	5.97	TRUE
Qwen3.6_FQ+dflash	35B-A3B	88	1		88	381	125.5	701.18	40.34	24.79	0	10	366	366	9.071	TRUE
Qwen3.6_FQ+dflash	35B-A3B	93	1		93	499	129.83	716.29	43.44	23.02	0	11	694	694	9.784	TRUE
Qwen3.6_FQ+dflash	35B-A3B	95	1		95	566	133.27	712.84	23.34	42.84	0	12	12	13	5.009	FALSE
Qwen3.6_FQ+dflash	35B-A3B	86	1		86	668	122.32	703.07	34.84	28.7	0	13	18	18	7.767	TRUE
Qwen3.6_FQ+dflash	35B-A3B	80	1		80	512	117.43	681.26	40.94	24.43	0	14	60	60	9.143	TRUE
Qwen3.6_FQ+dflash	35B-A3B	122	1		122	365	153.7	793.74	39.85	25.09	0	15	125	125	8.902	TRUE
Qwen3.6_FQ+dflash	35B-A3B	81	1		81	324	119.5	677.83	29.52	33.88	0	16	230	230	6.48	TRUE
Qwen3.6_FQ+dflash	35B-A3B	86	1		86	387	125.08	687.54	35.72	28	0	17	57500	57500	7.898	TRUE
Qwen3.6_FQ+dflash	35B-A3B	57	1		57	229	102.97	553.58	30.42	32.87	0	18	7	7	6.735	TRUE
Qwen3.6_FQ+dflash	35B-A3B	93	1		93	470	131.1	709.39	36.18	27.64	0	19	6	6	7.966	TRUE
Qwen3.6_FQ+dflash	35B-A3B	91	1		91	576	128.56	707.83	30.27	33.04	0	20	15	15	6.621	TRUE
Qwen3.6_FQ+dflash	35B-A3B	74	1		74	262	112.97	655.03	27.75	36.03	0	21	14	14	6.093	TRUE
Qwen3.6_FQ+dflash	35B-A3B	83	1		83	195	121.33	684.06	37.76	26.49	0	22	7	7	8.478	TRUE
Qwen3.6_FQ+dflash	35B-A3B	68	1		68	222	106.26	639.94	36.83	27.15	0	23	8	8	8.222	TRUE
Qwen3.6_FQ+dflash	35B-A3B	67	1		67	458	104.65	640.21	31.27	31.98	0	24	26	26	6.836	TRUE
Qwen3.6_FQ+dflash	35B-A3B	95	1		95	414	134.5	706.34	32.22	31.03	0	25	2	2	7.138	TRUE
Qwen3.6_FQ+dflash	35B-A3B	97	1		97	607	134.39	721.76	43.09	23.21	0	26	243	243	9.635	TRUE
Qwen3.6_FQ+dflash	35B-A3B	85	1		85	374	124.49	682.77	31.01	32.25	0	27	16	16	6.8	TRUE
Qwen3.6_FQ+dflash	35B-A3B	78	1		78	381	116.35	670.4	28.69	34.86	0	28	25	25	6.35	TRUE
Qwen3.6_FQ+dflash	35B-A3B	97	1		97	289	131.49	737.68	33.88	29.52	0	29	104	104	7.605	TRUE
Qwen3.6_FQ+dflash	35B-A3B	67	1		67	400	104.6	640.51	39.14	25.55	0	30	109	109	8.889	TRUE
Qwen3.6_FQ+dflash	35B-A3B	91	1		91	306	126.49	719.44	35.56	28.12	0	31	80	80	8.053	TRUE
Qwen3.6_FQ+dflash	35B-A3B	69	1		69	261	107.36	642.69	30.88	32.39	0	32	35	35	6.868	TRUE

Qwen3-VL-8B (VLM GPTQ，GPTQ INT4 (W4A16))

启动：

scripts/ci/musa/start_qwen3vl_server.sh

运行 benchmark：

MODEL_KEY=qwen3vl scripts/ci/musa/run_tilelang_benchmark.sh

Model	Size	PrefillLen(tokens)	Ctx(K)	Batch	NChip	Input(tokens)	Output(tokens)	Prefill(tps)	TTFT(ms)	Decode(tps)	TPOT(ms)	Profile	Prefix(tokens)	MeasuredOutput(tokens)
Qwen3VL	8b	128	16	1	1	128	16	97.865	1307.918	13.784	72.548	128:128	0	16
Qwen3VL	8b	250	16	1	1	250	16	168.405	1484.517	13.643	73.297	250:250	0	16
Qwen3VL	8b	500	16	1	1	500	16	260.2	1921.6	13.334	74.999	500:500	0	16
Qwen3VL	8b	1000	16	1	1	1000	16	337.692	2961.278	12.507	79.952	1000:1000	0	16
Qwen3VL	8b	2000	16	1	1	2000	16	291.99	6849.544	11.336	88.217	2000:2000	0	16
Qwen3VL	8b	250	16	1	1	500	16	125.607	1990.334	13.982	71.52	250:500	250	16
Qwen3VL	8b	250	16	1	1	1000	16	122.023	2048.789	12.308	81.251	250:1000	750	16
Qwen3VL	8b	250	16	1	1	2000	16	114.392	2185.475	11.521	86.801	250:2000	1750	16
Qwen3VL	8b	250	16	1	1	4000	16	118.897	2102.668	10.255	97.513	250:4000	3750	16
Qwen3VL	8b	250	16	1	1	8000	16	95.558	2616.218	8.349	119.774	250:8000	7750	16
Qwen3VL	8b	250	16	1	1	16000	16	67.205	3719.956	6.004	166.55	250:16000	15750	16

Model	Size	PrefillLen(k)	Ctx(k)	Batch	NChip	Input(k)	Output(k)	Prefill(tps)	Decode(tps)	Vision(fps)	Precision
Qwen3-VL	8B	0.25	16	1	1	0.25	0.05	168.711	13.638	0	w4a16
Qwen3-VL	8B	0.25	16	1	1	0.5	0.05	134.008	13.322	0	w4a16
Qwen3-VL	8B	0.25	16	1	1	1	0.05	124.818	12.999	0	w4a16
Qwen3-VL	8B	0.25	16	1	1	2	0.05	116.076	11.41	0	w4a16
Qwen3-VL	8B	0.25	16	1	1	4	0.05	116.694	10.04	0	w4a16
Qwen3-VL	8B	0.25	16	1	1	8	0.05	95.602	8.255	0	w4a16
Qwen3-VL	8B	0.25	16	1	1	16	0.05	67.272	5.997	0	w4a16

Qwen3.5-35B-A3B (MoE， W4A8_INT8)

启动：

MODEL_PATH=~/models/Qwen3.5-35B-A3B-QQQ-W4A8 ./scripts/ci/musa/start_qwen35_fq_server.sh --quantization qqq

运行 benchmark：

MODEL_KEY=qwen35_fq BENCH_PREFILL_SPECS='128:128, 256:256, 512:512, 1024:1024, 2048:2048' OUTPUT_TOKENS=100 bash scripts/ci/musa/run_tilelang_benchmark.sh

Model	Size	PrefillLen(tokens)	Ctx(K)	Batch	NChip	Input(tokens)	Output(tokens)	Prefill(tps)	TTFT(ms)	Decode(tps)	TPOT(ms)	Profile	Prefix(tokens)	MeasuredOutput(tokens)
Qwen3.5_FQ	35b_a3b	128	16	1	1	128	16	220.843	579.598	21.86	45.746	128:128	0	16
Qwen3.5_FQ	35b_a3b	256	16	1	1	256	16	300.622	851.568	21.856	45.754	256:256	0	16
Qwen3.5_FQ	35b_a3b	512	16	1	1	512	16	369.402	1386.025	21.786	45.901	512:512	0	16
Qwen3.5_FQ	35b_a3b	1024	16	1	1	1024	16	421.577	2428.975	21.644	46.203	1024:1024	0	16
Qwen3.5_FQ	35b_a3b	2048	16	1	1	2048	16	400.733	5110.635	21.51	46.49	2048:2048	0	16

Qwen3.6-35B-A3B (MoE， W4A8_INT8)

启动：

MODEL_PATH=~/models/Qwen3.6-35B-A3B-QQQ-W4A8 ./scripts/ci/musa/start_qwen36_fq_server.sh --quantization qqq

运行 benchmark：

MODEL_KEY=qwen36_fq BENCH_PREFILL_SPECS='128:128, 256:256, 512:512, 1024:1024, 2048:2048' OUTPUT_TOKENS=100 bash scripts/ci/musa/run_tilelang_benchmark.sh

Model	Size	PrefillLen(tokens)	Ctx(K)	Batch	NChip	Input(tokens)	Output(tokens)	Prefill(tps)	TTFT(ms)	Decode(tps)	TPOT(ms)	Profile	Prefix(tokens)	MeasuredOutput(tokens)
Qwen3.6_FQ	35b_a3b	128	16	1	1	128	100	216.11	592.292	21.61	46.274	128:128	0	100
Qwen3.6_FQ	35b_a3b	256	16	1	1	256	100	300.196	852.776	21.57	46.361	256:256	0	100
Qwen3.6_FQ	35b_a3b	512	16	1	1	512	100	362.556	1412.196	21.516	46.478	512:512	0	100
Qwen3.6_FQ	35b_a3b	1024	16	1	1	1024	100	413.023	2479.281	21.401	46.727	1024:1024	0	100
Qwen3.6_FQ	35b_a3b	2048	16	1	1	2048	100	393.216	5208.328	21.231	47.102	2048:2048	0	100