使用手册
摩尔线程数据中心GPU管理接口 (dcgmi)
摩尔线程数据中心GPU管理器（MTDCGM）包含用户开箱即用的命令行工具DCGMI，它是一款可以在数据中心环境管理和监控GPU资源的软件。安装完成后，用户不需要做太多的操作就可以直接使用，以下是DCGMI的使用帮助。
环境依赖
mt-dcgm v1.1.5 版本支持以下平台。
| 处理器 | 处理器架构 | 操作系统 | GPU | 驱动和MUSA Toolkit版本 | MTML版本 | MTBIOS版本 |
|---|---|---|---|---|---|---|
| Intel | X86_64 | Ubuntu 22.04.4 | Mthreads Server GPU | MUSA SDK 4.3.x and musa sdk 5.1.x | 2.2.x | S4000 3.4.3 S5000 4.3.13 |
| Intel | X86_64 | Alinux | debian | openEuler 22.03 | Mthreads Server GPU | MUSA SDK 4.3.x | 2.2.x | S5000 4.3.13 |
| AMD | X86_64 | Ubuntu 22.04.4 | Mthreads Server GPU | MUSA SDK 4.3.x | 2.2.x | S5000 4.3.13 |
dcgmi版本信息
dcgmi version: v1.1.5
community version: 3.3.6
依赖
dcgmi依赖mt-hostengine后台服务，使用时，确保mt-hostengine已在后台运行或者有使用dcgm api实现的后台服务运行。
如果需要运行诊断功能，则需要配置DCGM和MUSA toolkit环境变量，诊断功能需要在运行mt-hostengine之前配置该环境变量
export LD_LIBRARY_PATH=/usr/local/musa/lib:${LD_LIBRARY_PATH}
export PATH=/usr/local/musa/bin:${PATH}
export PATH=/usr/local/mtdcgm/bin:${PATH}
export LD_LIBRARY_PATH=/usr/local/mtdcgm/lib/x86_64-linux-gnu:${LD_LIBRARY_PATH}
命令参考
mt-hostengine使用命令
mt-hostengine作为MTDCGM的后台服务程序，为dcgmi或者其他需要连接到服务的程序提供后台服务。使用方法可参考如下的帮助命令：
$ mt-hostengine -h
mt-hostengine -t -p <PORT> -d <SOCKET_PATH> -n -b <IP_ADDRESS> --pid <FILENAME> --log-level
<LEVEL> -f <FILENAME> --log-rotate --denylist-modules <MODULEID[,MODULEID...]>
--service-account <USERNAME> --home-dir <Diagnostic home> -- --version -h
Where:
[-t | --term]: Terminates Host Engine [Best Effort]
[-p | --port]: Specify the port for the Host Engine
[-d | --domain-socket]: Specify the Unix domain socket path for host engine.
No TCP listening port is opened when this option is specified.
[-n | --no-daemon]: Tell the host engine not to daemonize on start-up
[-b | --bind-interface]: Specify the IP address of the network interface that the host engine should listen on.
ALL = bind to all interfaces.
Default: 127.0.0.1.
[--pid]: Specify the PID filename mt-hostengine should use to ensure that only one instance
is running.
[--log-level]: Specify the logging level. Default: ERROR
NONE - Disable logging
FATAL - Set log level to FATAL only
ERROR - Set log level to ERROR and above
WARN - Set log level to WARNING and above
INFO - Set log level to INFO and above
DEBUG - Set log level to DEBUG and above
[-f | --log-filename]: Specify the filename mt-hostengine should use to dump logging information.
Default: /var/log/mt-hostengine.log
[--log-rotate]: Rotate the log file if the log file with the same name already exists.
[--denylist-modules]: DCGM modules that should be added to the Denylist so they aren't run by the hostengine.
Pass a comma-separated list of module IDs like 1,2,3.
Module IDs are available in dcgm_structs.h as DcgmModuleId constants.
[--service-account]: Service account that will be used for unprivileged processes.
Usually this should be set to the default mthreads-dcgm account
[--home-dir]: The full path to the directory that the DCGM diagnostic should be launched from. This
becomes the default output directory for DCGM diagnostic log and stats
files.
[-- | --ignore_rest]: Ignores the rest of the labeled arguments following this flag.
[--version]: Displays version information and exits.
[-h | --help]: Displays usage information and exits.
dcgmi使用帮助
dcgmi的使用帮助信息不依赖后台服务。如果需要查看subsystem功能，则需要以在dcgmi后面接subsystem，然后再接-h查看具体的使用说明。dcgmi的使用帮助命令如下：
$ dcgmi -h
Usage:
dcgmi subsystem
dcgmi -v Flags:
-v --vv Get DCGMI version information
subsystem The desired subsystem to be accessed.
Subsystems Available:
diag System Validation/Diagnostic [dcgmi diag –h for more info]
health Health Monitoring [dcgmi health –h for more info]
group GPU Group Management [dcgmi group –h for more info]
fieldgroup Field Group Management [dcgmi fieldgroup –h for more info]
discovery Discover GPUs on the system [dcgmi discovery –h for more
info]
dmon Stats Monitoring of GPUs [dcgmi dmon –h for more info]
-- ignore_rest Ignores the rest of the labeled arguments following this
flag.
--version Displays version information and exits.
-h --help Displays usage information and exits.
诊断功能使用帮助
诊断是dcgmi的一个子模块，用于诊断系统和GPU的健康状态，使用方法如下：
dcgmi diag -h
diag -- Used to run diagnostics on the system.
Usage: dcgmi diag
dcgmi diag --host <IP/FQDN> -g <groupId> -r <diag> -p
<test_name.variable_name=variable_value> -c
</full/path/to/config/file> -i <gpuList> -v
--statsonfail --debugLogFile <debug file> --statspath <plugin
statistics path> -j --throttle-mask <> --fail-early
--check-interval <failure check interval> --iterations <iterations>
Flags:
-g --group groupId The group ID to query.
--host IP/FQDN Connects to specified IP or fully-qualified domain
name. To connect to a host engine that was
started with -d (unix socket), prefix the unix
socket filename with 'unix://'. [default =
localhost]
-h --help Displays usage information and exits.
-r --run diag Run a diagnostic. (Note: higher numbered tests
include all beneath.)
1 - Quick (System Validation ~ seconds)
2 - Medium (Extended System Validation ~ 2
minutes)
3 - Long (System HW Diagnostics ~ 15 minutes)
4 - Extended (Longer-running System HW
Diagnostics)
Specific tests to run may be specified by name,
and multiple tests may be specified as a comma
separated list.
-p --parameters test_name.variable_name=variable_value Test parameters to set
for this run.
-c --configfile /full/path/to/config/file Path to the configuration file.
-i --gpuList gpuList A comma-separated list of the gpus on which the
diagnostic should run. Cannot be used with -g.
-v --verbose Show information and warnings for each test.
--statsonfail Only output the statistics files if there was a
failure
--debugLogFiledebug file Encrypted log file for debug information. If a
debug level has been specified then "mtvs.log" is
the default log file.
--statspath plugin statistics path Write the plugin statistics to the
given path rather than the current
directory
-d --debugLevel debug level Debug level (One of NONE, FATAL, ERROR, WARN,
INFO, DEBUG, VERB). Default: DEBUG. The logfile
can be specified by the --debugLogFile
parameter.
-j --json Print the output in a json format
--train Run the diagnostic iteratively and generate a
configuration file of golden values based on
those results.
--throttle-mask Specify which throttling reasons should be
ignored. You can provide a comma separated list
of reasons.
--fail-early Enable early failure checks for the Targeted Stress
tests. When enabled, these tests check for a
failure once every 5 seconds (can be modified by
the --check-interval parameter) while the test is
running instead of a single check performed after
the test is complete. Disabled by default.
--check-intervalfailure check interval Specify the interval (in seconds)
at which the early failure checks should occur
for the Targeted Stress tests when early failure
checks are enabled. Default is once every 5
seconds. Interval must be between 1 and 300
--iterations iterations Specify a number of iterations of the diagnostic
to run consecutively. (Must be greater than 0.)
-- --ignore_rest Ignores the rest of the labeled arguments
following this flag.
举例：在GPU 0和1上检测系统配置是否正确
dcgmi diag -r 1 -i 0,1
使用远程检测
dcgmi diag -r 1 -i 0,1 --host 192.168.xx.xx:6666
自定义参数的用法（多个参数之间需要使用\;进行分割）
dcgmi diag -r 4 -i 0,1 -p memtest.test0=true\;memtest.test7=false
GPU Health使用帮助
Health是dcgmi的一个子模块，用于管理系统上GPU或者GPU组的健康状态，可以在不影响GPU程序运行的情况下进行监测，使用方法如下：
$ dcgmi health -h
health -- Used to manage the health watches of a group. The health of the GPUs
in a group can then be monitored during a process.
Usage: dcgmi health
dcgmi health --host <IP/FQDN> -g <groupId> -c -j
dcgmi health --host <IP/FQDN> -g <groupId> -f -j
dcgmi health --host <IP/FQDN> -g <groupId> -s <flags> -j -m <seconds> -u
<seconds>
Flags:
-g --group groupId The GPU group to query on the specified host.
--host IP/FQDN Connects to specified IP or fully-qualified domain
name. To connect to a host engine that was
started with -d (unix socket), prefix the unix
socket filename with 'unix://'. [default =
localhost]
-f --fetch Fetch the current watch status.
-s --set flags Set the watches to be monitored.
a - all watches
p - PCIe watches (*)
l - MTLink watches (*)
(*) watch requires 60 sec before first query
--clear Disable all watches being monitored.
-c --check Check to see if any errors or warnings have
occurred in the currently monitored watches.
-h --help Displays usage information and exits.
-j --json Print the output in a json format
-m --max-keep-ageseconds How long DCGM should cache the samples in
seconds.
-u --update-intervalseconds How often DCGM should retrieve health from the
driver in seconds.
-- --ignore_rest Ignores the rest of the labeled arguments
following this flag.
举例：在GPU组0上设置观测PCIe的健康
dcgmi health -g 0 -s p
获取观测的模块的观测状态
dcgmi health -g 0 -f
观测配置模块的健康状态
dcgmi health -g 0 -c
GPU Group功能使用帮助
GPU Group是dcgmi的一个子模块，用于管理系统上GPU或者GPU组，Group可以用于其他模块，使用方法如下：
dcgmi group -h
group -- Used to create and maintain groups of GPUs. Groups of GPUs can then be
uniformly controlled through other DCGMI subsystems.
Usage: dcgmi group
dcgmi group --host <IP/FQDN> -l -j
dcgmi group --host <IP/FQDN> -c <groupName> --default
dcgmi group --host <IP/FQDN> -c <groupName> -a <entityId>
dcgmi group --host <IP/FQDN> -d <groupId>
dcgmi group --host <IP/FQDN> -g <groupId> -i -j
dcgmi group --host <IP/FQDN> -g <groupId> -a <entityId>
dcgmi group --host <IP/FQDN> -g <groupId> -r <entityId>
Flags:
--host IP/FQDN Connects to specified IP or fully-qualified domain
name. To connect to a host engine that was
started with -d (unix socket), prefix the unix
socket filename with 'unix://'. [default =
localhost]
-l --list List the groups that currently exist for a host.
-d --delete groupId Delete a group on the remote host.
-c --create groupName Create a group on the remote host.
-h --help Displays usage information and exits.
-i --info Display the information for the specified group
ID.
-r --remove entityId Remove device(s) from group. (csv gpuIds, or
entityIds like gpu:0)
-a --add entityId Add device(s) to group. (csv gpuIds or entityIds
similar to gpu:0)
--default Adds all available GPUs to the group being
created.
-j --json Print the output in a json format
-- --ignore_rest Ignores the rest of the labeled arguments
following this flag.
举例：创建带有GPU的Group
dcgmi group -c testgroup -a 0,1,2
列出已创建的group
dcgmi group -l
删除创建的Group（默认Group不能删除）
dcgmi group -d 2
已有Group添加GPU
dcgmi group -g 2 -a 3
已有Group中移除GPU
dcgmi group -g 2 -r 3
Field Group功能使用帮助
Field Group是dcgmi的一个子模块，用于管理Fields或者Fields组，使用方法如下：
dcgmi fieldgroup -h
fieldgroup -- Used to create and maintain groups of field IDs. Groups of field
IDs can then be uniformly controlled through other DCGMI subsystems.
Usage: dcgmi fieldgroup
dcgmi fieldgroup --host <IP/FQDN> -l -j
dcgmi fieldgroup --host <IP/FQDN> -c <fieldGroupName> -f <fieldIds>
dcgmi fieldgroup --host <IP/FQDN> -i -g <fieldGroupId> -j
dcgmi fieldgroup --host <IP/FQDN> -d -g <fieldGroupId>
Flags:
--host IP/FQDN Connects to specified IP or fully-qualified domain
name. To connect to a host engine that was
started with -d (unix socket), prefix the unix
socket filename with 'unix://'. [default =
localhost]
-l --list List the field groups that currently exist for a
host.
-i --info Display the information for the specified field
group ID.
-d --delete Delete a field group on the remote host.
-c --create fieldGroupName Create a field group on the remote host.
-h --help Displays usage information and exits.
-g --fieldgroup fieldGroupId The field group to query on the specified host.
-f --fieldids fieldIds Comma-separated list of the field ids to add to a
field group when creating a new one.
-j --json Print the output in a json format
-- --ignore_rest Ignores the rest of the labeled arguments
following this flag.
举例：创建带有Field id的Fieldgroup
dcgmi fieldgroup -c testfieldgroup -f 50,51,52
列出已创建的Fieldgroup
dcgmi fieldgroup -l
删除创建的Fieldgroup（默认Group不能删除）
dcgmi fieldgroup -d -g 4
GPU设备发现使用帮助
设备发现是dcgmi的一个子模块，用于发现系统上GPU设备，使用方法如下：
dcgmi discovery -h
discovery -- Used to discover and identify GPUs and their attributes.
Usage: dcgmi discovery
dcgmi discovery --host <IP/FQDN> -l
dcgmi discovery --host <IP/FQDN> -i <flags> -g <groupId> -v
Flags:
-g --group groupId The group ID to query.
--host IP/FQDN Connects to specified IP or fully-qualified domain
name. To connect to a host engine that was
started with -d (unix socket), prefix the unix
socket filename with 'unix://'. [default =
localhost]
-l --list List all GPUs discovered on the host.
-i --info flags Specify which information to return. [default =
atp]
a - device info
-h --help Displays usage information and exits.
--gpuid gpuId The GPU ID to query.
-v --verbose Display policy information per GPU.
-- --ignore_rest Ignores the rest of the labeled arguments
following this flag.
举例：列出host上的GPU以及其基本信息
dcgmi discovery -l
显示GPU组中的每个设备的信息
dcgmi discovery -i a -g 0 -v
Device monitor功能使用帮助
Device Monitor是dcgmi的一个子模块，用于监视系统上GPU或者GPU组的指标，使用方法如下：
dcgmi dmon -h
dmon -- Used to monitor GPUs and their stats.
Usage: dcgmi dmon
dcgmi dmon -i <gpuId> -g <groupId> -f <fieldGroupId> -e <fieldId> -d
<delay> -c <count> -l
Flags:
--host IP/FQDN Connects to specified IP or fully-qualified domain
name. To connect to a host engine that was
started with -d (unix socket), prefix the unix
socket filename with 'unix://'. [default =
localhost]
-f --field-group-idfieldGroupId The field group to query on the specified
host.
-e --field-id fieldId Field identifier to view/inject.
-l --list List to look up the long names, short names and
field ids.
-h --help Displays usage information and exits.
-i --gpu-id gpuId The comma separated list of GPU/GPU-I/GPU-CI IDs
to run the dmon on. Default is -1 which runs for
all supported GPU. Run dcgmi discovery -c to
check list of available GPU entities
-g --group-id groupId The group to query on the specified host.
-d --delay delay In milliseconds. Integer representing how often
to query results from DCGM and print them for all
of the entities. [default = 1000 msec, Minimum
value = 1000 msec.]
-c --count count Integer representing How many times to loop
before exiting. [default- runs forever.]
-- --ignore_rest Ignores the rest of the labeled arguments
following this flag.
举例：在指定Group上观测Fieldgroup的数据5次
dcgmi dmon -g 0 -f 4 -c 5
一直打印指定Device的指定Field 数据
dcgmi dmon -i 0 -e 50,51,52
