跳到主要内容

MTDCGM v1.1.8 版本说明

环境要求​

处理器处理器架构操作系统GPUMUSA SDK 版本MTML 版本MTBIOS 版本
Intelx86_64Ubuntu 22.04.4
Alinux
Debian
OpenEuler 22.03
VesselOS 2.0
Moore Threads Server GPU4.3.82.3.5S4000 3.4.3
S5000 4.3.44 或更高版本
AMDx86_64Ubuntu 22.04.4Moore Threads Server GPU4.3.82.3.5S4000 3.4.3
S5000 4.3.44 或更高版本

更新日志​

新增功能​

  • 支持MPC模式下查询 Parent 设备的 Metrics,列表如下:
    • DCGM_FI_DRIVER_VERSION 1
    • DCGM_FI_MTML_VERSION 2
    • DCGM_FI_PROCESS_NAME 3
    • DCGM_FI_DEV_COUNT 4
    • DCGM_FI_MUSA_DRIVER_VERSION 5 (内部字段,不能对外)
    • DCGM_FI_DEV_NAME 50
    • DCGM_FI_DEV_BRAND 51
    • DCGM_FI_DEV_MTML_INDEX 52
    • DCGM_FI_DEV_SERIAL 53
    • DCGM_FI_DEV_UUID 54
    • DCGM_FI_DEV_PCI_BUSID 57
    • DCGM_FI_DEV_PCI_COMBINED_ID 58
    • DCGM_FI_DEV_PCI_SUBSYS_ID 59
    • DCGM_FI_DEV_MPC_MODE 67
    • DCGM_FI_DEV_CPU_AFFINITY_0 70
    • DCGM_FI_DEV_CPU_AFFINITY_1 71
    • DCGM_FI_DEV_CPU_AFFINITY_2 72
    • DCGM_FI_DEV_CPU_AFFINITY_3 73
    • DCGM_FI_DEV_MTBIOS_VERSION 85
    • DCGM_FI_DEV_MEM_AFFINITY_0 86
    • DCGM_FI_DEV_MEM_AFFINITY_1 87
    • DCGM_FI_DEV_MEM_AFFINITY_2 88
    • DCGM_FI_DEV_MEM_AFFINITY_3 89
    • DCGM_FI_DEV_GPU_CLOCK 100
    • DCGM_FI_DEV_MEM_CLOCK 101
    • DCGM_FI_DEV_MEMORY_TEMP 140
    • DCGM_FI_DEV_GPU_TEMP 150
    • DCGM_FI_DEV_POWER_USAGE 155
    • DCGM_FI_DEV_SLOWDOWN_TEMP 158
    • DCGM_FI_DEV_SHUTDOWN_TEMP 159
    • DCGM_FI_DEV_PCIE_TX_THROUGHPUT 200
    • DCGM_FI_DEV_PCIE_RX_THROUGHPUT 201
    • DCGM_FI_DEV_PCIE_REPLAY_COUNTER 202
    • DCGM_FI_DEV_GPU_UTIL 203
    • DCGM_FI_DEV_XID_ERRORS 230
    • DCGM_FI_DEV_PCIE_MAX_LINK_GEN 235
    • DCGM_FI_DEV_PCIE_MAX_LINK_WIDTH 236
    • DCGM_FI_DEV_PCIE_LINK_GEN 237
    • DCGM_FI_DEV_PCIE_LINK_WIDTH 238
    • DCGM_FI_DEV_FB_TOTAL 250
    • DCGM_FI_DEV_FB_FREE 251
    • DCGM_FI_DEV_FB_USED 252
    • DCGM_FI_DEV_FB_USED_PERCENT 254
    • DCGM_FI_DEV_MTLINK_CRC_DATA_ERROR_COUNT_TOTAL 419
    • DCGM_FI_DEV_MTLINK_REPLAY_ERROR_COUNT_TOTAL 429
    • DCGM_FI_DEV_MTLINK_RECOVERY_ERROR_COUNT_TOTAL 439
    • DCGM_FI_DEV_VIRTUAL_MODE 500
    • DCGM_FI_DEV_VOLTAGE 2000 (内部字段,不能对外)
    • DCGM_FI_PROF_MTLINK_TX_BYTES 2100
    • DCGM_FI_PROF_MTLINK_TX_BYTES 2150
    • DCGM_FI_DEV_GPU_TEMP_SENSOR1 2202
    • DCGM_FI_DEV_GPU_TEMP_SENSOR2 2203
    • DCGM_FI_DEV_ON_DIE_ECC_MODE_CURRENT 2204
    • DCGM_FI_DEV_EDC_MODE_CURRENT 2205
    • DCGM_FI_DEV_HWR_COUNTER 3000
    • DCGM_FI_DEV_MEMIRY_STATUS 3102
    • DCGMI_FI_DEV_MTLINK_UNCORRECTABLE_COUNTER_TOTAL 3200
    • DCGMI_FI_DEV_MTLINK_CORRECTABLE_COUNTER_TOTAL 3201
    • DCGMI_FI_DEV_MTLINK_BER_L0 - DCGMI_FI_DEV_MTLINK_BER_L13 (3203-3216)

已修复问题​

  • 修复 Ctrl + C 时 Profiling Metrics 数据采集线程未停止问题。

已知问题​

  • 使用 Profiling Metrics 时,模型训练收敛仍可能出现约 1% 的波动。
  • Profiling Metrics 功能在以下场景会出现冲突:
    • 使用 MTDCGM 多进程
    • MTDCGM 和 Moore Perf Compute 同时使用

相关文档​

  • 快速开始:根据当前版本的环境要求安装并验证 MTDCGM。
  • 功能概述:查看当前版本主要功能的常见用法。
  • FAQ:排查安装、连接、监控、诊断和集成过程中的常见问题。