Deepseek V4 模型部署

本章节包含 Deepseek-V4-Flash-FP8 的部署指导。

注意

注意 DeepSeek-V4 需要使用以下单独的镜像，不能使用 sglang:v0.5.6.post3 统一镜像。

# Intel/AMD 平台
docker pull registry.mthreads.com/devtech/dsv4-flash:1.9.3

PD 分离部署

Prefill 节点负责预填充，Decode 节点负责解码。二者通过 Mooncake 与 InfiniBand 传输 KV；最前端的 Router 统一对外暴露 HTTP 接口。DeepSeek-V4-Flash 的 prefill 和 Decode 实例各用1台机器，是 PD 分离部署最小单元。 但可根据上下文长度和性能要求，适当增加 Prefill 实例的数量，使得 Prefill 和 Decode 的吞吐相匹配。

Client → Router (:8192) → Prefill (:23456) / Decode (:23457)

Prefill启动

prefill启动脚本 start_prefill.sh

#!/bin/bash
# DeepSeek V4 MUSA PD Disaggregation - Prefill Node (统一启动脚本)
#
# Usage:
#   bash start_disagg_prefill.sh [MODEL_PATH]
#
# Example:
#   bash start_disagg_prefill.sh
#   bash start_disagg_prefill.sh /data/models/DeepSeek-V4-Flash-FP8

set -euo pipefail

# 自动获取本机 IP（优先走默认路由的网卡地址）
get_host_ip() {
    ip -4 route get 1.1.1.1 2>/dev/null | awk '{for (i = 1; i <= NF; i++) if ($i == "src") { print $(i + 1); exit }}'
}
HOST_IP=$(get_host_ip)
if [[ -z "${HOST_IP}" ]]; then
    HOST_IP=$(hostname -I | awk '{print $1}')
fi
if [[ -z "${HOST_IP}" ]]; then
    echo "ERROR: 无法自动获取本机 IP" >&2
    exit 1
fi

MODEL_PATH="${1:-/data/models/DeepSeek-V4-Flash-FP8}"
PREFILL_IP="${HOST_IP}:29500"

INIT_EXPERT_LOCATION="/workspace/eplb_profile/127k.pt"
DEEP_EP_CONFIG='{"normal_dispatch":{"num_sms":60,"num_max_nvl_chunked_send_tokens":26},"normal_combine":{"num_sms":60,"num_max_nvl_chunked_send_tokens":16}}'

echo "HOST_IP=${HOST_IP}  MODEL_PATH=${MODEL_PATH}  INIT_EXPERT=${INIT_EXPERT_LOCATION}"

# IB 设备自动检测（200G / 400G）
IB_DEVS=()
for dev in /sys/class/infiniband/*; do
    ibdev=$(basename "$dev")
    for port in "$dev"/ports/*; do
        rate=$(cat "$port/rate" 2>/dev/null)
        if [[ "$rate" == *"200 Gb"* || "$rate" == *"400 Gb"* ]]; then
            IB_DEVS+=("$ibdev")
            break
        fi
    done
done
IB_DEVS_SORTED=$(printf "%s\n" "${IB_DEVS[@]}" | sort -V | uniq)
SGLANG_IB_DEVICES=$(echo "$IB_DEVS_SORTED" | paste -sd, -)
export MCCL_IB_HCA="$SGLANG_IB_DEVICES"

export SGLANG_DSV4_FP4_EXPERTS=0
export SGLANG_OPT_USE_FUSED_HASH_TOPK=1
export SGLANG_OPT_USE_JIT_KERNEL_FUSED_TOPK=1
export SGLANG_JIT_DEEPGEMM_PRECOMPILE=0
export SGLANG_DSV4_MUSA_MOE_EXPERIMENTAL=1
export SGLANG_OPT_USE_JIT_INDEXER_METADATA=1
export SGLANG_OPT_USE_JIT_NORM=0
export SGLANG_OPT_FLASHMLA_SPARSE_PREFILL=0

# MHC
export SGLANG_OPT_USE_TILELANG_MHC_PRE=1
export SGLANG_OPT_MHC_PRE_BACKEND=auto
export SGLANG_OPT_DEEPGEMM_HC_PRENORM=1
export SGLANG_OPT_DEEPGEMM_HC_PRENORM_SPLIT_K=32
export SGLANG_OPT_MHC_PRE_BIG_FUSE_THREADS=0
export SGLANG_OPT_MHC_PRE_BIG_FUSE_HIDDEN_BLOCK=0
export SGLANG_OPT_MHC_PRE_BIG_FUSE_PASS_CONFIG=auto
export SGLANG_OPT_MHC_PRENORM_BACKEND=deepgemm
export SGLANG_OPT_MHC_PRENORM_SPLIT_K=32
export SGLANG_OPT_MHC_PRENORM_TILELANG_IMPL=auto
export SGLANG_OPT_USE_TILELANG_MHC_POST=1
export SGLANG_OPT_MHC_POST_THREADS=0
export SGLANG_OPT_MHC_POST_HIDDEN_BLOCK=0
export SGLANG_OPT_MHC_POST_PASS_CONFIG=auto
export SGLANG_OPT_MHC_POST_LAYOUT=auto
export SGLANG_OPT_MHC_POST_DIRECT_STORE=0

# GEMM / HC head
export SGLANG_OPT_BF16_FP32_GEMM_ALGO=deep_gemm
export SGLANG_OPT_HC_HEAD_TILELANG=1

# TopK
export SGLANG_DEEPSEEK_V4_MUSA_ENABLE_JIT_TOPK512=1

# RoPE / Hadamard
export SGLANG_OPT_DSV4_MUSA_TILELANG_ROPE=0
export SGLANG_OPT_DSV4_MUSA_FUSED_ROPE_HADAMARD=1
export SGLANG_OPT_DSV4_MUSA_FUSED_NSA_ROPE_HADAMARD=1

# Quant
export SGLANG_OPT_USE_TILEKERNELS_FP8_QUANT=1
export SGLANG_OPT_USE_TILEKERNELS_SWIGLU_QUANT=1

# MTP
export SGLANG_PREP_IN_CUDA_GRAPH=0
export SGLANG_DSV4_EAGLE_DRAFT_EXTEND_CUDA_GRAPH=0
export SGLANG_DSV4_HC_HEAD_TP=0

export SGLANG_DEEP_GEMM_BLOCK_M=256

# KV transfer
export SGLANG_DISAGGREGATION_QUEUE_SIZE=32
export SGLANG_DISAGGREGATION_THREAD_POOL_SIZE=64

# c128 online
export SGLANG_OPT_USE_ONLINE_COMPRESS=1
export SGLANG_DISAGGREGATION_ALL_CP_RANKS_TRANSFER=1

#export SGLANG_EXPERT_DISTRIBUTION_RECORDER_DIR=/workspace/working/eplb_profile

export SGLANG_HEALTH_CHECK_TIMEOUT=1200

python -m sglang.launch_server \
    --trust-remote-code \
    --model-path "$MODEL_PATH" \
    --tp 8 \
    --ep 8 \
    --mem-fraction-static 0.8 \
    --host 0.0.0.0 \
    --port 23456 \
    --chunked-prefill-size 69632 \
    --max-prefill-tokens 80000 \
    --disable-radix-cache \
    --disable-overlap-schedule \
    --log-level info \
    --watchdog-timeout 6000 \
    --disable-cuda-graph \
    --enable-nsa-prefill-context-parallel \
    --nsa-prefill-cp-mode round-robin-split \
    --moe-a2a-backend deepep \
    --deepep-mode normal \
    --deepep-config "$DEEP_EP_CONFIG" \
    --disaggregation-mode prefill \
    --disaggregation-transfer-backend mooncake \
    --disaggregation-bootstrap-port 8998 \
    --disaggregation-ib-device "${SGLANG_IB_DEVICES}" \
    --dist-init-addr "$PREFILL_IP" \
    --ep-num-redundant-experts 32 \
    --ep-dispatch-algorithm static \
    --init-expert-location "$INIT_EXPERT_LOCATION" \
    --eplb-algorithm deepseek \

32k 建议将chunked-prefill-size改为40960，max-prefill-tokens改为60000

---

Decode启动

decode启动脚本 start_decode.sh

#!/bin/bash
# DeepSeek V4 MUSA PD Disaggregation - Decode Node (Attn DP8 + MoE TP8)
#
# Usage:
#   bash start_disagg_decode.sh [MODEL_PATH]
#
# Example:
#   bash start_disagg_decode.sh
#   bash start_disagg_decode.sh /data/models/DeepSeek-V4-Flash-FP8

set -euo pipefail

# 自动获取本机 IP（优先走默认路由的网卡地址）
get_host_ip() {
    ip -4 route get 1.1.1.1 2>/dev/null | awk '{for (i = 1; i <= NF; i++) if ($i == "src") { print $(i + 1); exit }}'
}
HOST_IP=$(get_host_ip)
if [[ -z "${HOST_IP}" ]]; then
    HOST_IP=$(hostname -I | awk '{print $1}')
fi
if [[ -z "${HOST_IP}" ]]; then
    echo "ERROR: 无法自动获取本机 IP" >&2
    exit 1
fi

MODEL_PATH="${1:-/data/models/DeepSeek-V4-Flash-FP8}"
DECODER_IP="${HOST_IP}:29500"

echo "HOST_IP=${HOST_IP}  MODEL_PATH=${MODEL_PATH}"

# IB 设备自动检测（200G / 400G）
IB_DEVS=()
for dev in /sys/class/infiniband/*; do
    ibdev=$(basename "$dev")
    for port in "$dev"/ports/*; do
        rate=$(cat "$port/rate" 2>/dev/null)
        if [[ "$rate" == *"200 Gb"* || "$rate" == *"400 Gb"* ]]; then
            IB_DEVS+=("$ibdev")
            break
        fi
    done
done
IB_DEVS_SORTED=$(printf "%s\n" "${IB_DEVS[@]}" | sort -V | uniq)
SGLANG_IB_DEVICES=$(echo "$IB_DEVS_SORTED" | paste -sd, -)
export MCCL_IB_HCA="$SGLANG_IB_DEVICES"

export SGLANG_DSV4_FP4_EXPERTS=0
export SGLANG_OPT_USE_FUSED_HASH_TOPK=1
export SGLANG_OPT_USE_JIT_KERNEL_FUSED_TOPK=1
export SGLANG_JIT_DEEPGEMM_PRECOMPILE=1
export SGLANG_OPT_USE_JIT_EP_ACTIVATION=1

# MHC
export SGLANG_OPT_USE_TILELANG_MHC_PRE=1
export SGLANG_OPT_MHC_PRE_BACKEND=auto
export SGLANG_OPT_DEEPGEMM_HC_PRENORM=1
export SGLANG_OPT_DEEPGEMM_HC_PRENORM_SPLIT_K=32
export SGLANG_OPT_MHC_PRE_BIG_FUSE_THREADS=0
export SGLANG_OPT_MHC_PRE_BIG_FUSE_HIDDEN_BLOCK=0
export SGLANG_OPT_MHC_PRE_BIG_FUSE_PASS_CONFIG=auto
export SGLANG_OPT_MHC_PRENORM_BACKEND=deepgemm
export SGLANG_OPT_MHC_PRENORM_SPLIT_K=32
export SGLANG_OPT_MHC_PRENORM_TILELANG_IMPL=auto
export SGLANG_OPT_USE_TILELANG_MHC_POST=1
export SGLANG_OPT_MHC_POST_THREADS=0
export SGLANG_OPT_MHC_POST_HIDDEN_BLOCK=0
export SGLANG_OPT_MHC_POST_PASS_CONFIG=auto
export SGLANG_OPT_MHC_POST_LAYOUT=auto
export SGLANG_OPT_MHC_POST_DIRECT_STORE=0

# GEMM / HC head
export SGLANG_OPT_BF16_FP32_GEMM_ALGO=deep_gemm
export SGLANG_OPT_HC_HEAD_TILELANG=0

# TopK
export SGLANG_DEEPSEEK_V4_MUSA_ENABLE_JIT_TOPK512=1

# RoPE / Hadamard
export SGLANG_OPT_DSV4_MUSA_TILELANG_ROPE=0
export SGLANG_OPT_DSV4_MUSA_FUSED_ROPE_HADAMARD=1
export SGLANG_OPT_DSV4_MUSA_FUSED_NSA_ROPE_HADAMARD=1

# Quant
export SGLANG_OPT_USE_TILEKERNELS_FP8_QUANT=1
export SGLANG_OPT_USE_TILEKERNELS_SWIGLU_QUANT=1

# MTP（DP8 下 attention TP=1，hc_head 不做 TP 切分）
export SGLANG_PREP_IN_CUDA_GRAPH=0
export SGLANG_DSV4_EAGLE_DRAFT_EXTEND_CUDA_GRAPH=1
export SGLANG_DSV4_HC_HEAD_TP=0

export SGLANG_DEEP_GEMM_BLOCK_M=256

# KV transfer
export SGLANG_DISAGGREGATION_NUM_PRE_ALLOCATE_REQS=128

# c128 online
export SGLANG_OPT_USE_ONLINE_COMPRESS=1
export SGLANG_DISAGGREGATION_ALL_CP_RANKS_TRANSFER=1

python -m sglang.launch_server \
    --trust-remote-code \
    --model-path "$MODEL_PATH" \
    --tp 8 \
    --max-running-requests 144 \
    --num-reserved-decode-tokens 256 \
    --disaggregation-mode decode \
    --disaggregation-transfer-backend mooncake \
    --disaggregation-bootstrap-port 8998 \
    --disaggregation-ib-device "${SGLANG_IB_DEVICES}" \
    --dist-init-addr "$DECODER_IP" \
    --mem-fraction-static 0.9 \
    --swa-full-tokens-ratio 0.35 \
    --host 0.0.0.0 \
    --port 23457 \
    --cuda-graph-bs 1 2 4 6 8 10 12 14 16 24 32 40 48 56 64 72 80 88 96 104 112 120 128 136 144 \
    --chunked-prefill-size 8192 \
    --disable-radix-cache \
    --disable-overlap-schedule \
    --disable-custom-all-reduce \
    --skip-server-warmup \
    --speculative-algo EAGLE \
    --speculative-num-steps 3 \
    --speculative-eagle-topk 1 \
    --speculative-num-draft-tokens 4 \
    --dp 8 \
    --enable-dp-attention \

---

参考启动命令

以下示例命令启动 3个 Prefill 实例，1个 Decode 实例。如果只需要1个 Prefill 实例，则只需启动 prefill node 0，启动 Router 节点也只需第一个 --prefill 参数。

Prefill 节点：

# prefill node 0 执行
bash start_prefill.sh /data/models/DeepSeek-V4-Flash-FP8
# prefill node 1 执行
bash start_prefill.sh /data/models/DeepSeek-V4-Flash-FP8
# prefill node 2 执行
bash start_prefill.sh /data/models/DeepSeek-V4-Flash-FP8

Decode 节点：

# decode node 执行
bash start_decode.sh /data/models/DeepSeek-V4-Flash-FP8

Router 节点：

# prefill node 0 上执行
python3 -m sglang_router.launch_router   \
  --pd-disaggregation \
  --prefill <prefill_0_ip> 8998 \
  --prefill <prefill_1_ip> 8998 \
  --prefill <prefill_2_ip> 8998 \
  --decode <decode_ip> \
  --host 0.0.0.0 \
  --port 31000 \
  --prefill-policy round_robin \
  --worker-startup-timeout-secs 1200 \
  --health-check-timeout-secs 1200 \
  --prometheus-port 29001
# prefill_ip decode_ip示例：
# --prefill "http://10.20.32.81:23456" 8998 \
# --decode "http://10.20.32.83:23457" \

---

单机部署

以下脚本及命令是在单机8卡上混布 DeepSeek-V4-Flash，启动命令的参数说明如下。

位置参数	含义
$1	MODEL_PATH：模型权重路径
$2	EPLB_PATH：eplb profiling文件路径

---

服务启动脚本参考

服务启动脚本 start_server.sh

#!/bin/bash
# DeepSeek V4 MUSA PD Disaggregation - Prefill Node

# IB 设备自动检测（检测 200G/400G IB 设备）
IB_DEVS=()
for dev in /sys/class/infiniband/*; do
    ibdev=$(basename "$dev")
    for port in "$dev"/ports/*; do
        rate=$(cat "$port/rate" 2>/dev/null)
        if [[ "$rate" == *"200 Gb"* || "$rate" == *"400 Gb"* ]]; then
            IB_DEVS+=("$ibdev")
            break
        fi
    done
done
IB_DEVS_SORTED=$(printf "%s\n" "${IB_DEVS[@]}" | sort -V | uniq)
SGLANG_IB_DEVICES=$(echo "$IB_DEVS_SORTED" | paste -sd, -)
MCCL_IB_HCA="$SGLANG_IB_DEVICES"
export MCCL_IB_HCA

MODEL_PATH=$1
EPLB_PATH=$2

# 必要的环境变量（workaround 和功能配置）
export SGLANG_DSV4_FP4_EXPERTS=0
export SGLANG_ENABLE_JIT_DEEPGEMM=1
export SGLANG_OPT_USE_FUSED_HASH_TOPK=1
export SGLANG_JIT_DEEPGEMM_PRECOMPILE=0
export SGLANG_DSV4_MUSA_MOE_EXPERIMENTAL=1
export SGLANG_OPT_USE_JIT_KERNEL_FUSED_TOPK=1
export SGLANG_OPT_USE_JIT_INDEXER_METADATA=1
export SGLANG_OPT_USE_JIT_NORM=0
export SGLANG_OPT_FLASHMLA_SPARSE_PREFILL=0

# MHC
export SGLANG_OPT_USE_TILELANG_MHC_PRE=1
export SGLANG_OPT_MHC_PRE_BACKEND=auto
export SGLANG_OPT_DEEPGEMM_HC_PRENORM=1
export SGLANG_OPT_DEEPGEMM_HC_PRENORM_SPLIT_K=32
export SGLANG_OPT_MHC_PRE_BIG_FUSE_THREADS=0
export SGLANG_OPT_MHC_PRE_BIG_FUSE_HIDDEN_BLOCK=0
export SGLANG_OPT_MHC_PRE_BIG_FUSE_PASS_CONFIG=auto
export SGLANG_OPT_MHC_PRENORM_BACKEND=deepgemm
export SGLANG_OPT_MHC_PRENORM_SPLIT_K=32
export SGLANG_OPT_MHC_PRENORM_TILELANG_IMPL=auto
export SGLANG_OPT_USE_TILELANG_MHC_POST=1
export SGLANG_OPT_MHC_POST_THREADS=0
export SGLANG_OPT_MHC_POST_HIDDEN_BLOCK=0
export SGLANG_OPT_MHC_POST_PASS_CONFIG=auto
export SGLANG_OPT_MHC_POST_LAYOUT=auto
export SGLANG_OPT_MHC_POST_DIRECT_STORE=0

# GEMM / HC head
export SGLANG_OPT_BF16_FP32_GEMM_ALGO=deep_gemm
export SGLANG_OPT_HC_HEAD_TILELANG=1
export SGLANG_DEEP_GEMM_BLOCK_M=256


# TopK
export SGLANG_DEEPSEEK_V4_MUSA_ENABLE_JIT_TOPK512=1

# RoPE / Hadamard
export SGLANG_OPT_DSV4_MUSA_TILELANG_ROPE=0
export SGLANG_OPT_DSV4_MUSA_FUSED_ROPE_HADAMARD=1
export SGLANG_OPT_DSV4_MUSA_FUSED_NSA_ROPE_HADAMARD=1

# Quant
export SGLANG_OPT_USE_TILEKERNELS_FP8_QUANT=1
export SGLANG_OPT_USE_TILEKERNELS_SWIGLU_QUANT=1

#MTP
export SGLANG_PREP_IN_CUDA_GRAPH=0
export SGLANG_DSV4_EAGLE_DRAFT_EXTEND_CUDA_GRAPH=0
export SGLANG_DSV4_HC_HEAD_TP=0

export SGLANG_DISAGGREGATION_BOOTSTRAP_TIMEOUT=600

python -m sglang.launch_server \
    --trust-remote-code \
    --model-path "$MODEL_PATH" \
    --tp 8 \
    --ep 8 \
    --mem-fraction-static 0.8 \
    --host 0.0.0.0 \
    --port 24586 \
    --chunked-prefill-size 40960 \
    --max-prefill-tokens 60000 \
    --disable-overlap-schedule \
    --disable-custom-all-reduce \
    --log-level info \
    --tokenizer-backend fastokens \
    --watchdog-timeout 6000 \
    --enable-nsa-prefill-context-parallel \
    --nsa-prefill-cp-mode round-robin-split \
    --moe-a2a-backend deepep \
    --deepep-mode auto \
    --cuda-graph-max-bs 16 \
    --disable-radix-cache \
    --init-expert-location $EPLB_PATH \
    --ep-num-redundant-experts 32 \
    --eplb-algorithm deepseek \
    --ep-dispatch-algorithm static \

启动命令参考

以下命令所用的 eplb profiling 文件为示例，建议用户参考 EPLB profiling 章节，利用业务数据自行抓取。

# 40k场景
bash start_server.sh /data/models/DeepSeek-V4-Flash-FP8 /workspace/eplb_profile/seq40k.pt

# 20k场景
bash start_server.sh /data/models/DeepSeek-V4-Flash-FP8 /workspace/eplb_profile/seq20k.pt

如果不使用 EPLB，则启动命令不需要传递该参数，在start_server.sh 脚本中也删除以下参数：

--init-expert-location $EPLB_PATH \
--ep-num-redundant-experts 32 \
--eplb-algorithm deepseek \
--ep-dispatch-algorithm static \

---

EPLB profiling（如不使用可跳过本节）

当前场景 EPLB 是用的是 offline 算法, 拉起服务前需要预先抓取 eplb profiling 文件, 可以遵循本小节脚本来自行抓取.

服务侧

添加环境变量 SGLANG_EXPERT_DISTRIBUTION_RECORDER_DIR （例如，export SGLANG_EXPERT_DISTRIBUTION_RECORDER_DIR=/workspace/eplb_profile）来指定文件落盘路径, 同时启动脚本里面去掉如下参数：

--init-expert-location $EPLB_PATH \
--ep-num-redundant-experts 32 \
--eplb-algorithm deepseek \
--ep-dispatch-algorithm static \

再加上以下参数

--expert-distribution-recorder-mode stat \
--ep-num-redundant-experts 32 \
--ep-dispatch-algorithm static

请求侧

在服务侧拉起服务之后, 需要在请求侧curl发送抓取开始,结束和落盘信号。

# 1. 开始录制
curl -X POST http://127.0.0.1:24586/start_expert_distribution_record
# 2. 发送实际场景少量请求
bash bench.sh
# 3. 导出结果
curl -X POST http://127.0.0.1:24586/stop_expert_distribution_record
curl -X POST http://127.0.0.1:24586/dump_expert_distribution_record

随后可在请求侧指定的 SGLANG_EXPERT_DISTRIBUTION_RECORDER_DIR 路径看到抓取的pt文件, 之后在启动命令里使用该路径即可.

---