模型微调:基于DeepSpeed
1. 演练目标与架构
本节基于摩尔线程MTT S5000 GPU 完成 DeepSpeed 的微调,采用 BF16 精度。
2. 环境准备
机器信息
在 MTT S5000 上进行 DeepSpeed 微调。宿主机 Linux Driver 为 3.3.8,容器内 MUSA SDK 为 4.3.8。
使用摩尔线程预训练容器 training-suite:v2.1.7 ,容器内置 MUSA SDK 4.3.8、MT Megatron-LM 0.14.0、MT DeepSpeed 0.19.3、Torch-Musa 2.7.1。
创建并启动微调容器
# 创建容器
sudo docker create \
--privileged \
--env MTHREADS_VISIBLE_DEVICES=all \
--net host \
-v /data/:/data/ \
--name deepspeed \
registry.mthreads.com/mcconline/training-suite:v2.1.7 \
sleep infinity
# 启动并进入容器
sudo docker start deepspeed
sudo docker exec -it deepspeed bash
# 在容器里,启动ssh服务
service ssh restart
GPU 识别验证
进入容器后,验证MTT S5000 GPU 的识别状态与拓扑连接:
# 查看GPU硬件识别与状态
mthreads-gmi
3. 检查 DeepSpeed
在容器里,已经内置了 DeepSpeed 0.19.3。执行 ds_report:
# ds_report
[2026-02-28 15:47:47,251] [INFO] [real_accelerator.py:258:get_accelerator] Setting ds_accelerator to musa (auto detect)
[2026-02-28 15:47:48,310] [INFO] [logging.py:107:log_dist] [Rank -1] [TorchCheckpointEngine] Initialized with serialization = False
--------------------------------------------------
DeepSpeed C++/CUDA extension op report
--------------------------------------------------
NOTE: Ops not installed will be just-in-time (JIT) compiled at
runtime if needed. Op compatibility means that your system
meet the required dependencies to JIT install the op.
--------------------------------------------------
JIT compiled ops requires ninja
ninja .................. [OKAY]
--------------------------------------------------
op name ................ installed .. compatible
--------------------------------------------------
cpu_adam ............... [YES] ...... [OKAY]
musa_fused_adam ........ [YES] ...... [OKAY]
--------------------------------------------------
DeepSpeed general environment info:
torch install path ............... ['/usr/local/lib/python3.10/dist-packages/torch']
torch version .................... 2.7.1
deepspeed info ................... 0.17.2+350f1abf, 350f1abf, kuae-release-2.1
deepspeed wheel compiled w. ...... torch 2.7
shared memory (/dev/shm) size .... 1.00 GB