MTDCGM v1.1.8 版本说明
环境要求
| 处理器 | 处理器架构 | 操作系统 | GPU | MUSA SDK 版本 | MTML 版本 | MTBIOS 版本 |
|---|---|---|---|---|---|---|
| Intel | x86_64 | Ubuntu 22.04.4 Alinux Debian OpenEuler 22.03 VesselOS 2.0 | Moore Threads Server GPU | 4.3.8 | 2.3.5 | S4000 3.4.3 S5000 4.3.44 或更高版本 |
| AMD | x86_64 | Ubuntu 22.04.4 | Moore Threads Server GPU | 4.3.8 | 2.3.5 | S4000 3.4.3 S5000 4.3.44 或更高版本 |
更新日志
新增功能
- 支持MPC模式下查询 Parent 设备的 Metrics,列表如下:
- DCGM_FI_DRIVER_VERSION 1
- DCGM_FI_MTML_VERSION 2
- DCGM_FI_PROCESS_NAME 3
- DCGM_FI_DEV_COUNT 4
- DCGM_FI_MUSA_DRIVER_VERSION 5 (内部字段,不能对外)
- DCGM_FI_DEV_NAME 50
- DCGM_FI_DEV_BRAND 51
- DCGM_FI_DEV_MTML_INDEX 52
- DCGM_FI_DEV_SERIAL 53
- DCGM_FI_DEV_UUID 54
- DCGM_FI_DEV_PCI_BUSID 57
- DCGM_FI_DEV_PCI_COMBINED_ID 58
- DCGM_FI_DEV_PCI_SUBSYS_ID 59
- DCGM_FI_DEV_MPC_MODE 67
- DCGM_FI_DEV_CPU_AFFINITY_0 70
- DCGM_FI_DEV_CPU_AFFINITY_1 71
- DCGM_FI_DEV_CPU_AFFINITY_2 72
- DCGM_FI_DEV_CPU_AFFINITY_3 73
- DCGM_FI_DEV_MTBIOS_VERSION 85
- DCGM_FI_DEV_MEM_AFFINITY_0 86
- DCGM_FI_DEV_MEM_AFFINITY_1 87
- DCGM_FI_DEV_MEM_AFFINITY_2 88
- DCGM_FI_DEV_MEM_AFFINITY_3 89
- DCGM_FI_DEV_GPU_CLOCK 100
- DCGM_FI_DEV_MEM_CLOCK 101
- DCGM_FI_DEV_MEMORY_TEMP 140
- DCGM_FI_DEV_GPU_TEMP 150
- DCGM_FI_DEV_POWER_USAGE 155
- DCGM_FI_DEV_SLOWDOWN_TEMP 158
- DCGM_FI_DEV_SHUTDOWN_TEMP 159
- DCGM_FI_DEV_PCIE_TX_THROUGHPUT 200
- DCGM_FI_DEV_PCIE_RX_THROUGHPUT 201
- DCGM_FI_DEV_PCIE_REPLAY_COUNTER 202
- DCGM_FI_DEV_GPU_UTIL 203
- DCGM_FI_DEV_XID_ERRORS 230
- DCGM_FI_DEV_PCIE_MAX_LINK_GEN 235
- DCGM_FI_DEV_PCIE_MAX_LINK_WIDTH 236
- DCGM_FI_DEV_PCIE_LINK_GEN 237
- DCGM_FI_DEV_PCIE_LINK_WIDTH 238
- DCGM_FI_DEV_FB_TOTAL 250
- DCGM_FI_DEV_FB_FREE 251
- DCGM_FI_DEV_FB_USED 252
- DCGM_FI_DEV_FB_USED_PERCENT 254
- DCGM_FI_DEV_MTLINK_CRC_DATA_ERROR_COUNT_TOTAL 419
- DCGM_FI_DEV_MTLINK_REPLAY_ERROR_COUNT_TOTAL 429
- DCGM_FI_DEV_MTLINK_RECOVERY_ERROR_COUNT_TOTAL 439
- DCGM_FI_DEV_VIRTUAL_MODE 500
- DCGM_FI_DEV_VOLTAGE 2000 (内部字段,不能对外)
- DCGM_FI_PROF_MTLINK_TX_BYTES 2100
- DCGM_FI_PROF_MTLINK_TX_BYTES 2150
- DCGM_FI_DEV_GPU_TEMP_SENSOR1 2202
- DCGM_FI_DEV_GPU_TEMP_SENSOR2 2203
- DCGM_FI_DEV_ON_DIE_ECC_MODE_CURRENT 2204
- DCGM_FI_DEV_EDC_MODE_CURRENT 2205
- DCGM_FI_DEV_HWR_COUNTER 3000
- DCGM_FI_DEV_MEMIRY_STATUS 3102
- DCGMI_FI_DEV_MTLINK_UNCORRECTABLE_COUNTER_TOTAL 3200
- DCGMI_FI_DEV_MTLINK_CORRECTABLE_COUNTER_TOTAL 3201
- DCGMI_FI_DEV_MTLINK_BER_L0 - DCGMI_FI_DEV_MTLINK_BER_L13 (3203-3216)
已修复问题
- 修复 Ctrl + C 时 Profiling Metrics 数据采集线程未停止问题。
已知问题
- 使用 Profiling Metrics 时,模型训练收敛仍可能出现约 1% 的波动。
- Profiling Metrics 功能在以下场景会出现冲突:
- 使用 MTDCGM 多进程
- MTDCGM 和 Moore Perf Compute 同时使用

