跳到主要内容

GLM 模型部署

本章节以 GLM-5.2-FP8 部署为例，GLM-5.1-FP8 可使用相同方式，更改权重文件路径即可。

上下文规格（CTX）和部署方式对照​

CTX拓扑Prefill 脚本Decode 脚本 + 第 6 参数Prefill max-prefill / contextDecode context关键差异
64k及以下P2D4prefill_server_64k.shdecode_server.sh + 64k256k / 256k256k—
65k-128kP3D4prefill_server_128k.shdecode_server.sh + 128k256k / 256k256kALL_CP_RANKS_TRANSFER=1
129k-256kP3D4prefill_server_256k.shdecode_server.sh + 256k256k / 512k512kMC_SLICE_SIZE=4MB

硬性要求： Prefill / Decode 必须选同一 CTX 档。

128k 和 256k 的 Prefill 参数对比：

变量 / 参数128k256k
CONTEXT_LENGTH256k512k
MAX_PREFILL_TOKENS256k256k
CHUNKED_PREFILL_SIZE14k6k
PP_MAX_MICRO_BATCH_SIZE52
PREFILL_MEM_FRACTION_STATIC0.650.8
MC_SLICE_SIZE默认 4MB4MB
SGLANG_ALLOW_OVERWRITE_LONGER_CONTEXT_LEN—1

注意事项​

  • export SGLANG_ENABLE_OVERLAP_PLAN_STREAM=1 为实验特性，目的是将 decode 轮次内 forward 与前处理 overlap，追求 Decode 性能可使用，如追求长时间稳定性请删除。

  • 对于 PD 异构环境，增加了 trtllm attention backend（--dsa-prefill-backend / --dsa-decode-backend=trtllm）；PD 均为 S5000 时仍只能用 tilelang。

部署脚本​

脚本参数​

Prefill参数

bash prefill_server_<CTX>.sh <NODE_RANK> <PREFILL_MASTER_IP> <LOG_DIR> <MODEL_PATH> <HOST_IP>
参数含义
NODE_RANKPrefill 组内 rank，P0=0, P1=1, P2=2
PREFILL_MASTER_IPP0 的 IP（脚本内会拼 :4343 作为 dist-init-addr）
LOG_DIR日志目录（建议各组共用同一目录）
MODEL_PATH模型权重路径，如 /data/models/GLM-5.2-FP8
HOST_IP本机 IP（仅用于日志文件名）

注意：第 2 参只需 IP，不要写成 IP:24586。HTTP 服务端口固定 24586，分布式初始化端口固定 4343。

Decode参数

bash decode_server.sh <NODE_RANK> <DECODE_MASTER_IP> <LOG_DIR> <MODEL_PATH> <HOST_IP> <CTX_SIZE>
参数含义
NODE_RANKDecode 组内 rank，D0=0 … D3=3
DECODE_MASTER_IPD0 的 IP（脚本内拼 :4343）
CTX_SIZE64k / 128k / 256k（默认 128k）

Router参数

bash router.sh <PREFILL_MASTER_IP> <DECODE_MASTER_IP> <LOG_DIR>

Prefill 脚本 · 128k​

使用 PP3 CP8 EP8 并行策略。

prefill_server_128k.sh

# P节点
#!/bin/bash
# set -x
SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
export LD_LIBRARY_PATH=/usr/lib/x86_64-linux-gnu/:/usr/local/musa/lib:$LD_LIBRARY_PATH

pip uninstall -y sglang >/dev/null 2>&1 || true
export PYTHONPATH=/workspace/sglang/python:$PYTHONPATH

# Non-MTT stack: use sgl-kernel MUSA FP8 GEMM + tilelang DSA + DeepGEMM indexer
export SGLANG_USE_MTT_INDEXER=0
export SGLANG_USE_MTT_ATTN=0
export SGLANG_USE_MTT_F8GEMM=0
export SGLANG_USE_MTT_HGEMM=0
export SGLANG_USE_MTT_EPI_QKV=0
export SGLANG_NSA_FUSE_TOPK=1
export SGLANG_OPT_USE_TRITON_MASKED_SWIGLU_QUANT=1
export SGLANG_OPT_FUSED_INDEXER_QK_NORM_ROPE=1
export SGLANG_FUSED_MLA_ROPE_CACHE=1

export MCCL_SOCKET_IFNAME=bond0
export TP_SOCKET_IFNAME=bond0
export GLOO_SOCKET_IFNAME=bond0
export VLLM_PATCH_MUSA_CUSTOM_OPS=1
# export PYTORCH_MUSA_ALLOC_CONF=expandable_segments:True

# export MUSA_BLOCK_SCHEDULE_MODE=1
# export MUSA_USERQ=1
export SGLANG_DEEP_GEMM_BLOCK_M=256
export MUSA_PRINT_ENV=0
export MUSA_LAUNCH_BLOCKING=0
export MUSA_EXECUTION_TIMEOUT=900000
export DEEPEP_CPU_POLL_TIMEOUT=200
export MUSA_ERROR_DUMP_VERBOSE=1
export MUSA_ENABLE_LLC_OPT=1

export MCCL_PROTOS=2
export MCCL_IB_GID_INDEX=3
export MCCL_NET_SHARED_BUFFERS=0
export MCCL_IB_DISABLE=0
export MCCL_P2P_LEVEL=MT2
# export NVSHMEM_IB_TRAFFIC_CLASS=136

# mooncake
export MC_TE_METRIC=1
export MC_SLICE_SIZE="${MC_SLICE_SIZE:-$((4 * 1024 * 1024))}"
export MC_ENABLE_DEST_DEVICE_AFFINITY=1

export SGLANG_SET_CPU_AFFINITY=1
export SGLANG_DEEPEP_USE_MUSA_ACE=0
export SGLANG_SBO_COMBINE_SHARED=0
export SGLANG_DEEPEP_BF16_DISPATCH=0
export SGLANG_ENABLE_TORCH_INFERENCE_MODE=true
export SGLANG_ENABLE_SPEC_V2="${SGLANG_ENABLE_SPEC_V2:-true}"

export MATE_MUBIN_REPOSITORY=sw-compute-mate-mubin-generic-local
export MATE_DISABLE_VERSION_CHECK=1
# Mooncake KV transfer (prefill-side senders).
export SGLANG_DISAGGREGATION_QUEUE_SIZE=4
export SGLANG_DISAGGREGATION_THREAD_POOL_SIZE=8
export SGLANG_DISAGGREGATION_ALL_CP_RANKS_TRANSFER="${SGLANG_DISAGGREGATION_ALL_CP_RANKS_TRANSFER:-1}"
export SGLANG_DISAGGREGATION_BOOTSTRAP_TIMEOUT=600
export SGLANG_DISAGGREGATION_WAITING_TIMEOUT=600

# AUX metadata over TCP; avoids last-PP-stage AUX RDMA head-of-line blocking.
export SGLANG_MOONCAKE_SEND_AUX_TCP=1
export SGLANG_DISAGGREGATION_MAPPING_IB_DEVICE_TO_GPU=1
export SGLANG_HEALTH_CHECK_TIMEOUT=600

export SGLANG_FUSED_MLA_ROPE_CACHE=1
# export MUSA_EXECUTION_TIMEOUT=900000
export SGLANG_TORCH_PROFILER_DIR=$SCRIPT_DIR/traces
export SGLANG_EXPERT_DISTRIBUTION_RECORDER_DIR=$SCRIPT_DIR/traces
DEEP_EP_CONFIG="/workspace/code/deepep.config"


fp8_gemm_backend="auto"
if [ $SGLANG_USE_MTT_F8GEMM -gt 0 ]; then
  fp8_gemm_backend="mtt"
fi

nsa_backend="tilelang"
if [ $SGLANG_USE_MTT_ATTN -gt 0 ]; then
  nsa_backend="mtt_dsa"
fi

# IB 设备自动检测（200G / 400G）
IB_DEVS=()
for dev in /sys/class/infiniband/*; do
    ibdev=$(basename "$dev")
    for port in "$dev"/ports/*; do
        rate=$(cat "$port/rate" 2>/dev/null)
        state=$(cat "$port/state" 2>/dev/null)
        if [[ "$rate" == *"200 Gb"* || "$rate" == *"400 Gb"* ]] && [[ "$state" == "4"* ]]; then
            IB_DEVS+=("$ibdev")
            break
        fi
    done
done
IB_DEVS_SORTED=$(printf "%s\n" "${IB_DEVS[@]}" | sort -V | uniq)
SGLANG_IB_DEVICES=$(echo "$IB_DEVS_SORTED" | paste -sd, -)
export MCCL_IB_HCA="$SGLANG_IB_DEVICES"


NODE_RANK=$1
PREFILL_IP="${2}:4343"
LOG_DIR=$3
MODEL_PATH=$4
HOST_IP=$5
WORLD_SIZE=3
SGLANG_PORT=24586
MAX_PREFILL_TOKENS="${MAX_PREFILL_TOKENS:-$((256 * 1024))}"
CONTEXT_LENGTH="${CONTEXT_LENGTH:-$((256 * 1024))}"

mkdir -p "${LOG_DIR}"


PYTHON_ARGS=()

ENABLE_CP=1
if [ $ENABLE_CP -gt 0 ]; then
    PYTHON_ARGS+=(--dp-size 1)
    PYTHON_ARGS+=(--enable-nsa-prefill-context-parallel)
    sglang_dump_dir="saved_pp4cp8"
else
    PYTHON_ARGS+=(--dp-size 8)
    PYTHON_ARGS+=(--enable-dp-attention)
    sglang_dump_dir="saved_pp4dp8"
fi

if [[ "${PREFILL_DISABLE_EAGLE:-0}" != "1" ]]; then
    PYTHON_ARGS+=(--speculative-algorithm EAGLE)
    PYTHON_ARGS+=(--speculative-num-steps 5)
    PYTHON_ARGS+=(--speculative-eagle-topk 1)
    PYTHON_ARGS+=(--speculative-num-draft-tokens 6)
fi

export SGLANG_OPT_FLASHMLA_SPARSE_PREFILL=0
export SGLANG_DUMPER_ENABLE=0
export SGLANG_DUMPER_DIR=$sglang_dump_dir

# export SGLANG_PP_LAYER_PARTITION="40,38"
export SGLANG_PP_LAYER_PARTITION="${SGLANG_PP_LAYER_PARTITION:-28,26,24}"

RADIX_CACHE_ARGS=()
if [[ "${PREFILL_DISABLE_RADIX_CACHE:-0}" == "1" ]]; then
    RADIX_CACHE_ARGS+=(--disable-radix-cache)
fi

echo "${PYTHON_ARGS[@]}"


nohup python3 -u -m sglang.launch_server \
    --model $MODEL_PATH \
    --trust-remote-code \
    "${RADIX_CACHE_ARGS[@]}" \
    --disable-overlap-schedule \
    --enable-single-batch-overlap \
    --disable-cuda-graph \
    "${PYTHON_ARGS[@]}" \
    --tp-size 8 \
    --ep-size 8 \
    --pp-size $WORLD_SIZE \
    --moe-dense-tp-size 1 \
    --enable-dp-lm-head \
    --fp8-gemm-backend $fp8_gemm_backend \
    --attention-backend dsa \
    --dsa-prefill-backend tilelang \
    --dsa-decode-backend tilelang \
    --kv-cache-dtype fp8_e4m3 \
    --moe-a2a-backend deepep \
    --deepep-mode normal \
    --deepep-config "$DEEP_EP_CONFIG" \
    --sampling-backend flashinfer \
    --mem-fraction-static "${PREFILL_MEM_FRACTION_STATIC:-0.65}" \
    --max-running-requests $((16)) \
    --chunked-prefill-size "${CHUNKED_PREFILL_SIZE:-$((14 * 1024))}" \
    --max-prefill-tokens "${MAX_PREFILL_TOKENS}" \
    --context-length "${CONTEXT_LENGTH}" \
    --prefill-max-requests 4 \
    --pp-max-micro-batch-size "${PP_MAX_MICRO_BATCH_SIZE:-5}" \
    --dist-init-addr ${PREFILL_IP} \
    --nnodes ${WORLD_SIZE} \
    --node-rank ${NODE_RANK} \
    --host 0.0.0.0 \
    --port ${SGLANG_PORT} \
    --log-requests-level 3 \
    --log-level info \
    --tokenizer-backend fastokens \
    --enable-metrics \
    --enable-metrics-for-all-schedulers \
    --export-metrics-to-file \
    --export-metrics-to-file-dir ${LOG_DIR} \
    --enable-cache-report \
    --load-balance-method follow_bootstrap_room \
    --disaggregation-mode prefill \
    --watchdog-timeout 900 \
    --disaggregation-ib-device "${SGLANG_IB_DEVICES}" > "${LOG_DIR}/P${NODE_RANK}_${HOST_IP}_$(date '+%Y%m%d_%H%M%S').log" 2>&1 &

此处 Prefill 侧所有 PP 节点的 --dist-init-addr 均指向 P0 IP:4343。

Prefill 脚本 · 256k​

使用 PP3 CP8 EP8 并行策略，在 128k 脚本基础上，为 256k prompt 拉大 context、收紧 chunk / micro-batch，并加大 Mooncake RDMA slice。

prefill_server_256k.sh

# P节点 — 256k context
#!/bin/bash
# set -x
SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
export LD_LIBRARY_PATH=/usr/lib/x86_64-linux-gnu/:/usr/local/musa/lib:$LD_LIBRARY_PATH

pip uninstall -y sglang >/dev/null 2>&1 || true
export PYTHONPATH=/workspace/sglang/python:$PYTHONPATH

# Non-MTT stack: use sgl-kernel MUSA FP8 GEMM + tilelang DSA + DeepGEMM indexer
export SGLANG_USE_MTT_INDEXER=0
export SGLANG_USE_MTT_ATTN=0
export SGLANG_USE_MTT_F8GEMM=0
export SGLANG_USE_MTT_HGEMM=0
export SGLANG_USE_MTT_EPI_QKV=0
export SGLANG_NSA_FUSE_TOPK=1
export SGLANG_OPT_USE_TRITON_MASKED_SWIGLU_QUANT=1
export SGLANG_OPT_FUSED_INDEXER_QK_NORM_ROPE=1
export SGLANG_FUSED_MLA_ROPE_CACHE=1

export MCCL_SOCKET_IFNAME=bond0
export TP_SOCKET_IFNAME=bond0
export GLOO_SOCKET_IFNAME=bond0
export VLLM_PATCH_MUSA_CUSTOM_OPS=1
# export PYTORCH_MUSA_ALLOC_CONF=expandable_segments:True

# export MUSA_BLOCK_SCHEDULE_MODE=1
# export MUSA_USERQ=1
export SGLANG_DEEP_GEMM_BLOCK_M=256
export MUSA_PRINT_ENV=0
export MUSA_LAUNCH_BLOCKING=0
export MUSA_EXECUTION_TIMEOUT=900000
export DEEPEP_CPU_POLL_TIMEOUT=200
export MUSA_ERROR_DUMP_VERBOSE=1
export MUSA_ENABLE_LLC_OPT=1

export MCCL_PROTOS=2
export MCCL_IB_GID_INDEX=3
export MCCL_NET_SHARED_BUFFERS=0
export MCCL_IB_DISABLE=0
export MCCL_P2P_LEVEL=MT2
# export NVSHMEM_IB_TRAFFIC_CLASS=136

# mooncake
export MC_TE_METRIC=1
export MC_ENABLE_DEST_DEVICE_AFFINITY=1

export SGLANG_SET_CPU_AFFINITY=1
export SGLANG_DEEPEP_USE_MUSA_ACE=0
export SGLANG_SBO_COMBINE_SHARED=0
export SGLANG_DEEPEP_BF16_DISPATCH=0
export SGLANG_ENABLE_TORCH_INFERENCE_MODE=true
export SGLANG_ENABLE_SPEC_V2=true

export MATE_MUBIN_REPOSITORY=sw-compute-mate-mubin-generic-local
export MATE_DISABLE_VERSION_CHECK=1
# Mooncake KV transfer (prefill-side senders).
export SGLANG_DISAGGREGATION_QUEUE_SIZE=4
export SGLANG_DISAGGREGATION_THREAD_POOL_SIZE=8
export SGLANG_DISAGGREGATION_BOOTSTRAP_TIMEOUT=600
export SGLANG_DISAGGREGATION_WAITING_TIMEOUT=600

# Admit a 256K-token prompt and reserve room for generated tokens.
export MAX_PREFILL_TOKENS=$((256 * 1024))
export CONTEXT_LENGTH=$((512 * 1024))
export CHUNKED_PREFILL_SIZE=$((6 * 1024))
export PP_MAX_MICRO_BATCH_SIZE=2
export PREFILL_MEM_FRACTION_STATIC="${PREFILL_MEM_FRACTION_STATIC:-0.8}"
export SGLANG_ALLOW_OVERWRITE_LONGER_CONTEXT_LEN=1
export SGLANG_DISAGGREGATION_ALL_CP_RANKS_TRANSFER=1
export MC_SLICE_SIZE=$((4 * 1024 * 1024))

# AUX metadata over TCP; avoids last-PP-stage AUX RDMA head-of-line blocking.
export SGLANG_MOONCAKE_SEND_AUX_TCP=1
export SGLANG_DISAGGREGATION_MAPPING_IB_DEVICE_TO_GPU=1
export SGLANG_HEALTH_CHECK_TIMEOUT=600

export SGLANG_FUSED_MLA_ROPE_CACHE=1
# export MUSA_EXECUTION_TIMEOUT=900000
export SGLANG_TORCH_PROFILER_DIR=$SCRIPT_DIR/traces
export SGLANG_EXPERT_DISTRIBUTION_RECORDER_DIR=$SCRIPT_DIR/traces
DEEP_EP_CONFIG="/workspace/code/deepep.config"


fp8_gemm_backend="auto"
if [ $SGLANG_USE_MTT_F8GEMM -gt 0 ]; then
  fp8_gemm_backend="mtt"
fi

nsa_backend="tilelang"
if [ $SGLANG_USE_MTT_ATTN -gt 0 ]; then
  nsa_backend="mtt_dsa"
fi

# IB 设备自动检测（200G / 400G）
IB_DEVS=()
for dev in /sys/class/infiniband/*; do
    ibdev=$(basename "$dev")
    for port in "$dev"/ports/*; do
        rate=$(cat "$port/rate" 2>/dev/null)
        state=$(cat "$port/state" 2>/dev/null)
        if [[ "$rate" == *"200 Gb"* || "$rate" == *"400 Gb"* ]] && [[ "$state" == "4"* ]]; then
            IB_DEVS+=("$ibdev")
            break
        fi
    done
done
IB_DEVS_SORTED=$(printf "%s\n" "${IB_DEVS[@]}" | sort -V | uniq)
SGLANG_IB_DEVICES=$(echo "$IB_DEVS_SORTED" | paste -sd, -)
export MCCL_IB_HCA="$SGLANG_IB_DEVICES"


NODE_RANK=$1
PREFILL_IP="${2}:4343"
LOG_DIR=$3
MODEL_PATH=$4
HOST_IP=$5
WORLD_SIZE=3
SGLANG_PORT=24586

mkdir -p "${LOG_DIR}"


PYTHON_ARGS=()

ENABLE_CP=1
if [ $ENABLE_CP -gt 0 ]; then
    PYTHON_ARGS+=(--dp-size 1)
    PYTHON_ARGS+=(--enable-nsa-prefill-context-parallel)
    sglang_dump_dir="saved_pp4cp8"
else
    PYTHON_ARGS+=(--dp-size 8)
    PYTHON_ARGS+=(--enable-dp-attention)
    sglang_dump_dir="saved_pp4dp8"
fi

export SGLANG_OPT_FLASHMLA_SPARSE_PREFILL=0
export SGLANG_DUMPER_ENABLE=0
export SGLANG_DUMPER_DIR=$sglang_dump_dir

# export SGLANG_PP_LAYER_PARTITION="40,38"
export SGLANG_PP_LAYER_PARTITION="28,26,24"

echo "${PYTHON_ARGS[@]}"


nohup python3 -u -m sglang.launch_server \
    --model $MODEL_PATH \
    --trust-remote-code \
    --disable-overlap-schedule \
    --enable-single-batch-overlap \
    --disable-cuda-graph \
    "${PYTHON_ARGS[@]}" \
    --tp-size 8 \
    --ep-size 8 \
    --pp-size $WORLD_SIZE \
    --moe-dense-tp-size 1 \
    --enable-dp-lm-head \
    --fp8-gemm-backend $fp8_gemm_backend \
    --attention-backend dsa \
    --dsa-prefill-backend tilelang \
    --dsa-decode-backend tilelang \
    --kv-cache-dtype fp8_e4m3 \
    --moe-a2a-backend deepep \
    --deepep-mode normal \
    --deepep-config "$DEEP_EP_CONFIG" \
    --sampling-backend flashinfer \
    --mem-fraction-static ${PREFILL_MEM_FRACTION_STATIC} \
    --max-running-requests $((16)) \
    --chunked-prefill-size ${CHUNKED_PREFILL_SIZE} \
    --max-prefill-tokens ${MAX_PREFILL_TOKENS} \
    --context-length ${CONTEXT_LENGTH} \
    --prefill-max-requests 4 \
    --pp-max-micro-batch-size ${PP_MAX_MICRO_BATCH_SIZE} \
    --dist-init-addr ${PREFILL_IP} \
    --nnodes ${WORLD_SIZE} \
    --node-rank ${NODE_RANK} \
    --host 0.0.0.0 \
    --port ${SGLANG_PORT} \
    --log-requests-level 3 \
    --log-level info \
    --tokenizer-backend fastokens \
    --enable-metrics \
    --enable-metrics-for-all-schedulers \
    --export-metrics-to-file \
    --export-metrics-to-file-dir ${LOG_DIR} \
    --enable-cache-report \
    --load-balance-method follow_bootstrap_room \
    --disaggregation-mode prefill \
    --watchdog-timeout 900 \
    --speculative-algorithm EAGLE \
    --speculative-num-steps 5 \
    --speculative-eagle-topk 1 \
    --speculative-num-draft-tokens 6 \
    --disaggregation-ib-device "${SGLANG_IB_DEVICES}" > "${LOG_DIR}/P${NODE_RANK}_${HOST_IP}_$(date '+%Y%m%d_%H%M%S').log" 2>&1 &

Prefill 脚本 · 64k​

使用 PP2 CP8 EP8 并行策略，SGLANG_PP_LAYER_PARTITION=40,38，适合更短上下文。

prefill_server_64k.sh

# P节点
#!/bin/bash
# set -x
SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
export LD_LIBRARY_PATH=/usr/lib/x86_64-linux-gnu/:/usr/local/musa/lib:$LD_LIBRARY_PATH

pip uninstall -y sglang >/dev/null 2>&1 || true
export PYTHONPATH=/workspace/sglang/python:$PYTHONPATH

# Non-MTT stack: use sgl-kernel MUSA FP8 GEMM + tilelang DSA + DeepGEMM indexer
export SGLANG_USE_MTT_INDEXER=0
export SGLANG_USE_MTT_ATTN=0
export SGLANG_USE_MTT_F8GEMM=0
export SGLANG_USE_MTT_HGEMM=0
export SGLANG_USE_MTT_EPI_QKV=0
export SGLANG_NSA_FUSE_TOPK=1
export SGLANG_OPT_USE_TRITON_MASKED_SWIGLU_QUANT=1
export SGLANG_OPT_FUSED_INDEXER_QK_NORM_ROPE=1
export SGLANG_FUSED_MLA_ROPE_CACHE=1

export MCCL_SOCKET_IFNAME=bond0
export TP_SOCKET_IFNAME=bond0
export GLOO_SOCKET_IFNAME=bond0
export VLLM_PATCH_MUSA_CUSTOM_OPS=1
# export PYTORCH_MUSA_ALLOC_CONF=expandable_segments:True

# export MUSA_BLOCK_SCHEDULE_MODE=1
# export MUSA_USERQ=1
export SGLANG_DEEP_GEMM_BLOCK_M=256
export MUSA_PRINT_ENV=0
export MUSA_LAUNCH_BLOCKING=0
export MUSA_EXECUTION_TIMEOUT=900000
export DEEPEP_CPU_POLL_TIMEOUT=200
export MUSA_ERROR_DUMP_VERBOSE=1
export MUSA_ENABLE_LLC_OPT=1

export MCCL_PROTOS=2
export MCCL_IB_GID_INDEX=3
export MCCL_NET_SHARED_BUFFERS=0
export MCCL_IB_DISABLE=0
export MCCL_P2P_LEVEL=MT2
# export NVSHMEM_IB_TRAFFIC_CLASS=136

# mooncake
export MC_TE_METRIC=1
export MC_ENABLE_DEST_DEVICE_AFFINITY=1

export SGLANG_SET_CPU_AFFINITY=1
export SGLANG_DEEPEP_USE_MUSA_ACE=0
export SGLANG_SBO_COMBINE_SHARED=0
export SGLANG_DEEPEP_BF16_DISPATCH=0
export SGLANG_ENABLE_TORCH_INFERENCE_MODE=true
export SGLANG_ENABLE_SPEC_V2=true

export MATE_MUBIN_REPOSITORY=sw-compute-mate-mubin-generic-local
export MATE_DISABLE_VERSION_CHECK=1
# Mooncake KV transfer (prefill-side senders).
export SGLANG_DISAGGREGATION_QUEUE_SIZE=4
export SGLANG_DISAGGREGATION_THREAD_POOL_SIZE=8
export SGLANG_DISAGGREGATION_ALL_CP_RANKS_TRANSFER=1
export SGLANG_DISAGGREGATION_BOOTSTRAP_TIMEOUT=600
export SGLANG_DISAGGREGATION_WAITING_TIMEOUT=600

# AUX metadata over TCP; avoids last-PP-stage AUX RDMA head-of-line blocking.
export SGLANG_MOONCAKE_SEND_AUX_TCP=1
export SGLANG_DISAGGREGATION_MAPPING_IB_DEVICE_TO_GPU=1
export SGLANG_HEALTH_CHECK_TIMEOUT=600

export SGLANG_FUSED_MLA_ROPE_CACHE=1
# export MUSA_EXECUTION_TIMEOUT=900000
export SGLANG_TORCH_PROFILER_DIR=$SCRIPT_DIR/traces
export SGLANG_EXPERT_DISTRIBUTION_RECORDER_DIR=$SCRIPT_DIR/traces
DEEP_EP_CONFIG="/workspace/code/deepep.config"


fp8_gemm_backend="auto"
if [ $SGLANG_USE_MTT_F8GEMM -gt 0 ]; then
  fp8_gemm_backend="mtt"
fi

nsa_backend="tilelang"
if [ $SGLANG_USE_MTT_ATTN -gt 0 ]; then
  nsa_backend="mtt_dsa"
fi

# IB 设备自动检测（200G / 400G）
IB_DEVS=()
for dev in /sys/class/infiniband/*; do
    ibdev=$(basename "$dev")
    for port in "$dev"/ports/*; do
        rate=$(cat "$port/rate" 2>/dev/null)
        state=$(cat "$port/state" 2>/dev/null)
        if [[ "$rate" == *"200 Gb"* || "$rate" == *"400 Gb"* ]] && [[ "$state" == "4"* ]]; then
            IB_DEVS+=("$ibdev")
            break
        fi
    done
done
IB_DEVS_SORTED=$(printf "%s\n" "${IB_DEVS[@]}" | sort -V | uniq)
SGLANG_IB_DEVICES=$(echo "$IB_DEVS_SORTED" | paste -sd, -)
export MCCL_IB_HCA="$SGLANG_IB_DEVICES"


NODE_RANK=$1
PREFILL_IP="${2}:4343"
LOG_DIR=$3
MODEL_PATH=$4
HOST_IP=$5
WORLD_SIZE=2
SGLANG_PORT=24586

mkdir -p "${LOG_DIR}"


PYTHON_ARGS=()

ENABLE_CP=1
if [ $ENABLE_CP -gt 0 ]; then
    PYTHON_ARGS+=(--dp-size 1)
    PYTHON_ARGS+=(--enable-nsa-prefill-context-parallel)
    sglang_dump_dir="saved_pp4cp8"
else
    PYTHON_ARGS+=(--dp-size 8)
    PYTHON_ARGS+=(--enable-dp-attention)
    sglang_dump_dir="saved_pp4dp8"
fi

export SGLANG_OPT_FLASHMLA_SPARSE_PREFILL=0
export SGLANG_DUMPER_ENABLE=0
export SGLANG_DUMPER_DIR=$sglang_dump_dir

export SGLANG_PP_LAYER_PARTITION="40,38"

echo "${PYTHON_ARGS[@]}"


nohup python3 -u -m sglang.launch_server \
    --model $MODEL_PATH \
    --trust-remote-code \
    --disable-overlap-schedule \
    --enable-single-batch-overlap \
    --disable-cuda-graph \
    "${PYTHON_ARGS[@]}" \
    --tp-size 8 \
    --ep-size 8 \
    --pp-size $WORLD_SIZE \
    --moe-dense-tp-size 1 \
    --enable-dp-lm-head \
    --fp8-gemm-backend $fp8_gemm_backend \
    --attention-backend dsa \
    --dsa-prefill-backend tilelang \
    --dsa-decode-backend tilelang \
    --kv-cache-dtype fp8_e4m3 \
    --moe-a2a-backend deepep \
    --deepep-mode normal \
    --deepep-config "$DEEP_EP_CONFIG" \
    --sampling-backend flashinfer \
    --sampling-backend flashinfer \
    --mem-fraction-static 0.80 \
    --max-running-requests $((36)) \
    --chunked-prefill-size $((8 * 1024)) \
    --max-prefill-tokens $((256 * 1024)) \
    --context-length $((256 * 1024)) \
    --dist-init-addr ${PREFILL_IP} \
    --nnodes ${WORLD_SIZE} \
    --node-rank ${NODE_RANK} \
    --host 0.0.0.0 \
    --port ${SGLANG_PORT} \
    --log-requests-level 3 \
    --log-level info \
    --tokenizer-backend fastokens \
    --enable-metrics \
    --enable-metrics-for-all-schedulers \
    --export-metrics-to-file \
    --export-metrics-to-file-dir ${LOG_DIR} \
    --enable-cache-report \
    --load-balance-method follow_bootstrap_room \
    --disaggregation-mode prefill \
    --watchdog-timeout 900 \
    --speculative-algorithm EAGLE \
    --speculative-num-steps 5 \
    --speculative-eagle-topk 1 \
    --speculative-num-draft-tokens 6 \
    --disaggregation-ib-device "${SGLANG_IB_DEVICES}" > "${LOG_DIR}/P${NODE_RANK}_${HOST_IP}_$(date '+%Y%m%d_%H%M%S').log" 2>&1 &

Decode 脚本​

使用 DP32 EP32 并行策略，同一套 Decode 脚本通过第 6 参 CTX_SIZE 适配 64k/128k/256k 上下文范围。其中设为 256k 时会自动设置以下参数：

  • --context-length 512k

  • MC_SLICE_SIZE=4MB

  • SGLANG_DISAGGREGATION_ALL_CP_RANKS_TRANSFER=1

  • 若 CONTEXT_LENGTH > SGLANG_DSA_MAX_SEQ_LEN_FOR_CAPTURE，自动抬高 capture 上限

decode_server.sh

# D节点
#!/bin/bash
# set -x
SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
export LD_LIBRARY_PATH=/usr/lib/x86_64-linux-gnu/:/usr/local/musa/lib:$LD_LIBRARY_PATH

export PYTHONPATH=/workspace/sglang/python:$PYTHONPATH
export SGLANG_NSA_FUSE_TOPK=1
export SGLANG_SYNC_AFTER_RUN_BATCH=0

# --- Operator fusions (validated / known-good on this DeepEP LL decode) ---
export SGLANG_OPT_USE_TRITON_MASKED_SWIGLU_QUANT=1
export SGLANG_OPT_USE_JIT_EP_ACTIVATION=1
export SGLANG_OPT_SWIGLU_CLAMP_FUSION=1
# Keep fused GEMV at tiny-M only (M<=3 == legacy <4); larger M is slower here
export SGLANG_MUSA_FUSED_GEMV_MAX_M=3
# TileKernels paths are prefill-oriented; leave off for decode
export SGLANG_OPT_USE_TILEKERNELS_FP8_QUANT=0
export SGLANG_OPT_USE_TILEKERNELS_SWIGLU_QUANT=0
export SGLANG_MASKED_GEMM_FAST_ACT=0
# Indexer / MLA rope fusions
export SGLANG_OPT_FUSED_INDEXER_QK_NORM_ROPE=1
export SGLANG_FUSED_MLA_ROPE_CACHE=1
export SGLANG_USE_FUSED_METADATA_COPY=1
# MUSA: Triton fused metadata copy (CUDA JIT unavailable).
# Cap MUST cover the longest prompt you serve. zhipu bench uses encoding=65536;
# 16384 truncated page_table → wrong KV → accept rate collapse (~0.60→~0.30).
# PREP_IN_CUDA_GRAPH gathers this width every decode step — 262144 was ~4x
# overkill vs observed max ~131k (p50 still ~65k). 139264 = 128k+11k margin.
export SGLANG_FUSED_METADATA_COPY_BACKEND=triton
export SGLANG_DSA_ENABLE_MTP_PRECOMPUTE_METADATA=1
export SGLANG_DSA_MAX_SEQ_LEN_FOR_CAPTURE=262144
export SGLANG_OPT_USE_JIT_NORM=1

export MCCL_SOCKET_IFNAME=bond0
export TP_SOCKET_IFNAME=bond0
export GLOO_SOCKET_IFNAME=bond0
export VLLM_PATCH_MUSA_CUSTOM_OPS=1

export MUSA_PRINT_ENV=0
export MUSA_LAUNCH_BLOCKING=0
# export MUSA_EXECUTION_TIMEOUT=12000
export MUSA_ERROR_DUMP_VERBOSE=1
export MUSA_ENABLE_LLC_OPT=1
export SGLANG_FUSE_VERIFY_SAMPLE=1
# Optional MUSA scheduler knobs (lwj experimented with these)
# BLOCK_SCHEDULE_MODE=0 breaks cuda-graph capture (MUSA op not permitted).
export MUSA_BLOCK_SCHEDULE_MODE=1
# USERQ=0: avoid leaving sample/VTG in user queue during host waits.
export MUSA_USERQ=0

export MCCL_PROTOS=2
export MCCL_IB_GID_INDEX=3
export MCCL_NET_SHARED_BUFFERS=0

# mooncake
export MC_TE_METRIC=0
export MC_SLICE_SIZE="${MC_SLICE_SIZE:-$((4 * 1024 * 1024))}"
export MC_ENABLE_DEST_DEVICE_AFFINITY=1

export SGLANG_SET_CPU_AFFINITY=1
export SGLANG_SBO_SHARED_DISPATCH_TWO_STREAM=1
# Retry combine/shared overlap alone (FAST_ACT stays off)
export SGLANG_SBO_COMBINE_SHARED=1
export SGLANG_DEEPEP_BF16_DISPATCH=0
export SGLANG_ENABLE_TORCH_INFERENCE_MODE=true
export SGLANG_ENABLE_SPEC_V2=true
export SGLANG_ALLOW_OVERWRITE_LONGER_CONTEXT_LEN=1
export NVSHMEM_IBGDA_FORCE_NIC_BUF_MEMTYPE=gpumem
export SGLANG_ENABLE_STRICT_MEM_CHECK_DURING_IDLE=0
export SGLANG_DISAGGREGATION_HEARTBEAT_INTERVAL=100
export SGLANG_DISAGGREGATION_QUEUE_SIZE=4
export SGLANG_DISAGGREGATION_THREAD_POOL_SIZE=8
export SGLANG_DISAGGREGATION_ALL_CP_RANKS_TRANSFER="${SGLANG_DISAGGREGATION_ALL_CP_RANKS_TRANSFER:-1}"
export SGLANG_DISAGGREGATION_BOOTSTRAP_TIMEOUT=1200
export SGLANG_DISAGGREGATION_WAITING_TIMEOUT=1200
export SGLANG_DISAGGREGATION_MAPPING_IB_DEVICE_TO_GPU=1
export SGLANG_HEALTH_CHECK_TIMEOUT=600
export SGLANG_PREP_IN_CUDA_GRAPH=1
export SGLANG_DSA_PREP_IN_CUDA_GRAPH=1
export SGLANG_ENABLE_OVERLAP_PLAN_STREAM=1
export SGLANG_DSA_EAGLE_DRAFT_EXTEND_CUDA_GRAPH=1
export SGLANG_LM_HEAD_TP_SIZE=8
# # KV transfer
export SGLANG_MOONCAKE_SEND_AUX_TCP=1
export SGLANG_DISAGGREGATION_NUM_PRE_ALLOCATE_REQS=128
export SGLANG_LM_HEAD_GRAPH_BS_LOG=0
export SGLANG_EAGLE_HANG_STAGE_LOG=0
export  SGLANG_OVERLAP_PUBLISH_WAIT_WARN_MS=5000
export MATE_MUBIN_REPOSITORY=sw-compute-mate-mubin-generic-local
export MATE_DISABLE_VERSION_CHECK=1
# init_new pageable metadata H2D host stall warn (ms); 0=off
export SGLANG_INIT_NEW_META_H2D_WARN_MS="${SGLANG_INIT_NEW_META_H2D_WARN_MS:-2000}"
#export SGLANG_USE_FUSED_METADATA_COPY=0
fp8_gemm_backend="auto"
nsa_backend="tilelang"
IB_DEVS=()

for dev in /sys/class/infiniband/*; do
    ibdev=$(basename "$dev")

    for port in "$dev"/ports/*; do
        # 速率（如 200 Gb/sec）
        rate=$(cat "$port/rate" 2>/dev/null)
        # link_layer: InfiniBand / Ethernet (RoCE)
        link=$(cat "$port/link_layer" 2>/dev/null)

        # 只要是 200G（IB 或 RoCE 都收）
        if [[ "$rate" == *"400 Gb"* ]]; then
            IB_DEVS+=("$ibdev")
            break
        fi
    done
done

# 排序 + 去重
IB_DEVS_SORTED=$(printf "%s\n" "${IB_DEVS[@]}" | sort -V | uniq)

# 生成 NCCL / SGLang 变量
NCCL_IB_HCA=$(echo "$IB_DEVS_SORTED" | paste -sd, -)
SGLANG_IB_DEVICES="$NCCL_IB_HCA"
echo "IB_DEVICES=$NCCL_IB_HCA"

ENABLE_PROFILER=0
POSITIONAL=()
while [[ $# -gt 0 ]]; do
    case $1 in
        --profiler)
            ENABLE_PROFILER=1
            shift
            ;;
        *)
            POSITIONAL+=("$1")
            shift
            ;;
    esac
done
set -- "${POSITIONAL[@]}"

NODE_RANK=$1
DECODER_IP="${2}:4343"
LOG_DIR=$3
MODEL_PATH=$4
HOST_IP=$5
CTX_SIZE="${6:-128k}"
WORLD_SIZE=4
SGLANG_PORT=23457
DEFAULT_MEM_FRACTION_STATIC=0.75
graph_bs=4

case "$CTX_SIZE" in
    64k|128k)
        CONTEXT_LENGTH=$((256 * 1024))
        ;;
    256k)
        CONTEXT_LENGTH=$((512 * 1024))
        # Match prefill_server_256k.sh KV transfer settings.
        export SGLANG_DISAGGREGATION_ALL_CP_RANKS_TRANSFER=1
        export MC_SLICE_SIZE=$((4 * 1024 * 1024))
        ;;
    *)
        echo "Unsupported CTX_SIZE: $CTX_SIZE" >&2
        exit 1
        ;;
esac

# The fused metadata-copy table must cover the long prompt on Decode nodes.
if (( CONTEXT_LENGTH > SGLANG_DSA_MAX_SEQ_LEN_FOR_CAPTURE )); then
    export SGLANG_DSA_MAX_SEQ_LEN_FOR_CAPTURE="${CONTEXT_LENGTH}"
fi

mkdir -p "${LOG_DIR}"

export SGLANG_DUMPER_ENABLE=0

# Leave headroom for EAGLE draft / draft-extend cuda-graph capture after the
# main model graph is captured. 0.8 left only ~3–5GB free and capture aborted
# with MUSA "operation not permitted when stream is capturing". Override via env.
MEM_FRACTION_STATIC="${MEM_FRACTION_STATIC:-$DEFAULT_MEM_FRACTION_STATIC}"

# Soft watchdog off by default. 60s false-fires during weight load / cuda-graph
# capture: Scheduler soft is active while is_initializing, and TokenizerManager
# also uses soft_watchdog_timeout → mass py-spy during startup.
# After server is ready, enable for soak, e.g.:
#   SOFT_WATCHDOG_TIMEOUT=120 ./decode_server.sh ...
# Hang-stage marks (h2d_kv_lens_*/mlp_ag_*/init_new_meta_h2d_*) need
# SGLANG_EAGLE_HANG_STAGE_LOG=1 (already exported above) to print enter/exit.
# Slow init_new metadata H2D: SGLANG_INIT_NEW_META_H2D_WARN_MS (default 2000).
SOFT_WATCHDOG_TIMEOUT="${SOFT_WATCHDOG_TIMEOUT:-}"
WATCHDOG_TIMEOUT="${WATCHDOG_TIMEOUT:-300}"
SOFT_WATCHDOG_ARGS=()
if [[ -n "${SOFT_WATCHDOG_TIMEOUT}" ]]; then
    SOFT_WATCHDOG_ARGS+=(--soft-watchdog-timeout "${SOFT_WATCHDOG_TIMEOUT}")
fi

PYTHON_ARGS=()
WITH_MTP=1
if [ $WITH_MTP -gt 0 ]; then
    # Known-good: steps=3, draft=4 (TPOT~35). 2/3 measured worse (~58).
    PYTHON_ARGS+=(--speculative-algorithm EAGLE)
    PYTHON_ARGS+=(--speculative-num-steps 5)
    PYTHON_ARGS+=(--speculative-eagle-topk 1)
    PYTHON_ARGS+=(--speculative-num-draft-tokens 6)
fi

echo "${PYTHON_ARGS[@]}"

if [ "$ENABLE_PROFILER" -gt 0 ]; then
    PROFILER_DIR="${LOG_DIR}/profiler/D${NODE_RANK}_${HOST_IP}"
    mkdir -p "$PROFILER_DIR"
    export SGLANG_TORCH_PROFILER_DIR="$PROFILER_DIR"
    export SGLANG_PROFILE_WITH_STACK=${SGLANG_PROFILE_WITH_STACK:-0}
    export SGLANG_PROFILE_RECORD_SHAPES=${SGLANG_PROFILE_RECORD_SHAPES:-1}
    echo "SGLANG_TORCH_PROFILER_DIR=$PROFILER_DIR"
    echo "Profiler: 就绪后手动 start，例如 curl -X POST http://127.0.0.1:${SGLANG_PORT}/start_profile -H 'Content-Type: application/json' -d '{\"start_step\":3,\"num_steps\":30}'"
fi


nohup python3 -u -m sglang.launch_server \
    --model $MODEL_PATH \
    --trust-remote-code \
    --disable-radix-cache \
    --enable-single-batch-overlap \
    --cuda-graph-bs $(seq 1 $graph_bs) \
    "${PYTHON_ARGS[@]}" \
    --tp-size $((WORLD_SIZE * 8)) \
    --ep-size $((WORLD_SIZE * 8)) \
    --dp-size $((WORLD_SIZE * 8)) \
    --enable-dp-attention \
    --moe-dense-tp-size 1 \
    --enable-dp-lm-head \
    --fp8-gemm-backend $fp8_gemm_backend \
    --attention-backend dsa \
    --dsa-prefill-backend tilelang \
    --dsa-decode-backend tilelang \
    --kv-cache-dtype fp8_e4m3 \
    --moe-a2a-backend deepep \
    --deepep-mode low_latency \
    --sampling-backend flashinfer \
    --mem-fraction-static "${MEM_FRACTION_STATIC}" \
    --max-running-requests $((WORLD_SIZE * 8 * graph_bs)) \
    --context-length "${CONTEXT_LENGTH}" \
    --reasoning-parser glm45 \
    --tool-call-parser glm47  \
    --dist-init-addr ${DECODER_IP} \
    --nnodes ${WORLD_SIZE} \
    --node-rank ${NODE_RANK} \
    --host 0.0.0.0 \
    --port ${SGLANG_PORT} \
    --log-requests-level 3 \
    --log-level info \
    "${SOFT_WATCHDOG_ARGS[@]}" \
    --watchdog-timeout ${WATCHDOG_TIMEOUT} \
    --enable-metrics \
    --enable-metrics-for-all-schedulers \
    --export-metrics-to-file \
    --export-metrics-to-file-dir ${LOG_DIR} \
    --load-balance-method total_requests \
    --disaggregation-mode decode  \
    ${SKIP_SERVER_WARMUP:+--skip-server-warmup} \
    --speculative-accept-threshold-single 1.0 \
    --speculative-accept-threshold-acc 1.0 \
    --disaggregation-ib-device $SGLANG_IB_DEVICES > "${LOG_DIR}/D${NODE_RANK}_${HOST_IP}_$(date '+%Y%m%d_%H%M%S').log" 2>&1 &

此处 Decode 侧所有节点的 --dist-init-addr 均指向 D0 IP:4343。

只有 D0 会打印 The server is fired up and ready to roll! 并对外提供 HTTP；D1–D3 日志停在 Dummy health check server started 属正常。

Router 脚本​

router.sh

# router
# source /root/.local/bin/uv-virtualenvwrapper.sh
# workon sglang-default
# sleep 120

PREFILL_IP="http://${1}:24586"
DECODE_IP="http://$2:23457"
LOG_DIR=$3


export LD_LIBRARY_PATH=/usr/lib/x86_64-linux-gnu/:/usr/local/musa/lib:$LD_LIBRARY_PATH
export PYTHONPATH=/workspace/sglang/python:$PYTHONPATH
export TORCH_DEVICE_BACKEND_AUTOLOAD=0

export RUST_BACKTRACE=1
ulimit -n 65535
nohup python -m sglang_router.launch_router \
    --pd-disaggregation \
    --prefill $PREFILL_IP \
    --decode $DECODE_IP \
    --host 0.0.0.0 \
    --request-timeout-secs 7200 \
    --prefill-policy round_robin \
    --decode-policy round_robin \
    --port 32000 > "${LOG_DIR}/router_$(date '+%Y%m%d_%H%M%S').log" 2>&1 &

PREFILL_IP / DECODE_IP 分别为 P0、D0 的 HTTP 地址（端口 24586 / 23457）。

参考启动命令​

假设：

  • Prefill：P0=<prefill_node0_ip>，另两台为 P1/P2

  • Decode：D0 + D1/D2/D3（共 4 台）

  • LOG_DIR=/data/logs/glm52_pd

  • MODEL=/data/models/GLM-5.2-FP8

先在各节点清理旧进程及端口：4343、24586、23457、32000、29000。

64k及以下（P2D4）​

bash prefill_server_64k.sh 0 <prefill_node0_ip> $LOG_DIR $MODEL <prefill_node0_ip>
bash prefill_server_64k.sh 1 <prefill_node0_ip> $LOG_DIR $MODEL <prefill_node1_ip>

bash decode_server.sh 0 <decode_node0_ip> $LOG_DIR $MODEL <decode_node0_ip> 64k
bash decode_server.sh 1 <decode_node0_ip> $LOG_DIR $MODEL <decode_node1_ip> 64k
bash decode_server.sh 2 <decode_node0_ip> $LOG_DIR $MODEL <decode_node2_ip> 64k
bash decode_server.sh 3 <decode_node0_ip> $LOG_DIR $MODEL <decode_node3_ip> 64k

65k-128k（P3D4）​

bash prefill_server_128k.sh 0 <prefill_node0_ip> $LOG_DIR $MODEL <prefill_node0_ip>
bash prefill_server_128k.sh 1 <prefill_node0_ip> $LOG_DIR $MODEL <prefill_node1_ip>
bash prefill_server_128k.sh 2 <prefill_node0_ip> $LOG_DIR $MODEL <prefill_node2_ip>

bash decode_server.sh 0 <decode_node0_ip> $LOG_DIR $MODEL <decode_node0_ip> 128k
bash decode_server.sh 1 <decode_node0_ip> $LOG_DIR $MODEL <decode_node1_ip> 128k
bash decode_server.sh 2 <decode_node0_ip> $LOG_DIR $MODEL <decode_node2_ip> 128k
bash decode_server.sh 3 <decode_node0_ip> $LOG_DIR $MODEL <decode_node3_ip> 128k

129k-256k（P3D4）​

bash prefill_server_256k.sh 0 <prefill_node0_ip> $LOG_DIR $MODEL <prefill_node0_ip>
bash prefill_server_256k.sh 1 <prefill_node0_ip> $LOG_DIR $MODEL <prefill_node1_ip>
bash prefill_server_256k.sh 2 <prefill_node0_ip> $LOG_DIR $MODEL <prefill_node2_ip>

bash decode_server.sh 0 <decode_node0_ip> $LOG_DIR $MODEL <decode_node0_ip> 256k
bash decode_server.sh 1 <decode_node0_ip> $LOG_DIR $MODEL <decode_node1_ip> 256k
bash decode_server.sh 2 <decode_node0_ip> $LOG_DIR $MODEL <decode_node2_ip> 256k
bash decode_server.sh 3 <decode_node0_ip> $LOG_DIR $MODEL <decode_node3_ip> 256k

Router（P0 上执行）​

bash router.sh <prefill_node0_ip> <decode_node0_ip> $LOG_DIR

就绪判定与压测注意​

  1. P0 / D0 日志出现：The server is fired up and ready to roll!

  2. D4 拓扑下 D1–D3：Dummy health check server started...

  3. Router 日志无 Address already in use（重点查 :32000、:29000）

  4. 健康检查：

curl http://<decode_node0_ip>:23457/health
curl http://<prefill_node0_ip>:24586/health

  1. 长上下文权重加载 + graph capture 更久，等 fired up 后再打压测。

  2. 调用请求访问 Router：http://<prefill_node0_ip>:32000。

版权所有 2024 摩尔线程，本文档受国际版权法保护。

摩尔线程和摩尔线程徽标是摩尔线程智能科技 (北京)有限责任公司的注册商标。

免责声明

本文档提供有关摩尔线程产品的信息。本文档并未授权任何知识产权的许可，并未以明示或暗示，或以禁止反言或其他方式授予任何知识产权许可。