KUAE 云原生套件使用手册

MT Container Toolkit 使用指南

摩尔线程容器运行时套件兼容多种容器引擎，包括Docker、containerd 和 CRI-O等。

Docker

摩尔线程容器套件提供了不同的选项来列举任何容器所支持的 GPU。本用户指南演示了摩尔线程容器套件的以下功能。

使用环境变量来启用以下功能：
- 枚举 GPU 并控制 GPU 对容器的可见性。
- 使用 MTHREADS_DRIVER_CAPABILITIES 控制驱动能力在容器中的可见性。

GPU 枚举

通过设置 MTHREADS_VISIBLE_DEVICES 环境变量可以指定特定的 GPU 设备到容器中。

MTHREADS_VISIBLE_DEVICES 可以设置的值如下：

可设置的值	描述
`0,1,2,` 或者 `438e3949-f6f2-f7b4-c904-8ffb9d6fa9d3`	通过 GPU 的设备编号或者 GPU UUID 指定。
`all`	所有的 GPU 设备均可见。
`void`, empty 或者 unset	`mthreads-container-runtime` 的行为将和 `runc<sup>`[1]`</sup>` 保持一致。

示例:

使用全部 GPU：

❯ docker run --rm -e MTHREADS_VISIBLE_DEVICES=all ubuntu:20.04 mthreads-gmi
Thu Jul 10 11:52:47 2025
---------------------------------------------------------------
    mthreads-gmi:2.0.0           Driver Version:3.0.0
---------------------------------------------------------------
ID   Name           |PCIe                |%GPU  Mem          
     Device Type    |Pcie Lane Width     |Temp  MPC Capable  
                                         |      ECC Mode     
+-------------------------------------------------------------+
0    MTT S4000      |00000000:01:00.0    |0%    518MiB(49152MiB)
     Physical       |16x(16x)            |89C   YES          
                                         |      N/A          
---------------------------------------------------------------

---------------------------------------------------------------
Processes:
ID   PID       Process name                         GPU Memory
                                                         Usage
+-------------------------------------------------------------+
   No running processes found
---------------------------------------------------------------

使用设备号指定 GPU：

❯ docker run --rm -e MTHREADS_VISIBLE_DEVICES=1 ubuntu:20.04 mthreads-gmi
Thu Jul 10 11:52:47 2025
---------------------------------------------------------------
    mthreads-gmi:2.0.0           Driver Version:3.0.0
---------------------------------------------------------------
ID   Name           |PCIe                |%GPU  Mem          
     Device Type    |Pcie Lane Width     |Temp  MPC Capable  
                                         |      ECC Mode     
+-------------------------------------------------------------+
0    MTT S4000      |00000000:01:00.0    |0%    518MiB(49152MiB)
     Physical       |16x(16x)            |89C   YES          
                                         |      N/A          
---------------------------------------------------------------

---------------------------------------------------------------
Processes:
ID   PID       Process name                         GPU Memory
                                                         Usage
+-------------------------------------------------------------+
   No running processes found
---------------------------------------------------------------

使用 GPU UUID 指定 GPU：

❯ mthreads-gmi --query | grep UUID
    GPU UUID                                      :  7e8581f5-6722-035b-9b7f-07b8ed270fd0
    GPU UUID                                      :  2716ae75-44e4-231b-df26-f7c2c4abb678

❯ docker run --rm -e MTHREADS_VISIBLE_DEVICES=2716ae75-44e4-231b-df26-f7c2c4abb678 ubuntu:20.04 mthreads-gmi
Thu Jul 10 11:52:47 2025
---------------------------------------------------------------
    mthreads-gmi:2.0.0           Driver Version:3.0.0
---------------------------------------------------------------
ID   Name           |PCIe                |%GPU  Mem          
     Device Type    |Pcie Lane Width     |Temp  MPC Capable  
                                         |      ECC Mode     
+-------------------------------------------------------------+
0    MTT S4000      |00000000:01:00.0    |0%    518MiB(49152MiB)
     Physical       |16x(16x)            |89C   YES          
                                         |      N/A          
---------------------------------------------------------------

---------------------------------------------------------------
Processes:
ID   PID       Process name                         GPU Memory
                                                         Usage
+-------------------------------------------------------------+
   No running processes found
---------------------------------------------------------------

驱动能力

MTHREADS_DRIVER_CAPABILITIES 用于控制容器中用户态驱动程序的能力。

MTHREADS_DRIVER_CAPABILITIES 可以设置的值如下：

可设置的值	描述
`compute,video` 或者 `graphics,utility`	可以传入多个驱动能力，以 “,” 为分割符。
`all`	GPU 所有的能力。
empty 或者 unset	设置默认值:`compute`, `utility`。
`void`	不挂载任何驱动能力。

支持设置的驱动能力列表。

驱动能力	描述
`compute`	使用 `MUSA` 和 `OpenCL` 的应用。
`graphics`	使用 `OpenGL` 和 `Vulkan` 的应用。
`utility`	使用 `mthreads-gmi` 和 `MTML`。
`video`	使用编解码能力的应用。

示例：

❯ docker run --rm \
      -e MTHREADS_VISIBLE_DEVICES=all \
      -e MTHREADS_DRIVER_CAPABILITIES=compute,utility \
      ubuntu:20.04 mthreads-gmi
Thu Jul 10 11:52:47 2025
---------------------------------------------------------------
    mthreads-gmi:2.0.0           Driver Version:3.0.0
---------------------------------------------------------------
ID   Name           |PCIe                |%GPU  Mem          
     Device Type    |Pcie Lane Width     |Temp  MPC Capable  
                                         |      ECC Mode     
+-------------------------------------------------------------+
0    MTT S4000      |00000000:01:00.0    |0%    518MiB(49152MiB)
     Physical       |16x(16x)            |89C   YES          
                                         |      N/A          
---------------------------------------------------------------

---------------------------------------------------------------
Processes:
ID   PID       Process name                         GPU Memory
                                                         Usage
+-------------------------------------------------------------+
   No running processes found
---------------------------------------------------------------

sGPU

MTHREADS_QOS_MEMORY_LIMIT 控制一个 sGPU 设备可以消耗的最大内存，单位是字节，MTHREADS_QOS_COMPUTING_POWER_WEIGHT 指定一个 sGPU 设备在分配计算能力时的权重。

注意：

MTHREADS_QOS_COMPUTING_POWER_WEIGHT 是可选的。默认值是 1。

在使用前面的环境变量之前，请确保 sGPU 内核模块已经正确安装和加载。

你需要将 MTHREADS_VISIBLE_DEVICES 和前面的环境变量一起指定。

下面的例子通过设置 MTHREADS_VISIBLE_DEVICES 为 0 来指定 GPU 0 为共享内存和计算能力的物理 GPU。MTHREADS_QOS_MEMORY_LIMIT=1073741824 表示 sGPU 设备可以消耗最多 1 GB 的内存。MTHREADS_QOS_COMPUTING_POWER_WEIGHT=2 表示 sGPU 设备在分配计算能力时的权重为 2。

$ docker run --rm \    
      -e MTHREADS_VISIBLE_DEVICES=0 \
      -e MTHREADS_QOS_MEMORY_LIMIT=1073741824 \
      -e MTHREADS_QOS_COMPUTING_POWER_WEIGHT=2 \
      ubuntu mthreads-gmi
Thu Jul 10 11:52:47 2025
---------------------------------------------------------------
    mthreads-gmi:2.0.0           Driver Version:3.0.0
---------------------------------------------------------------
ID   Name           |PCIe                |%GPU  Mem          
     Device Type    |Pcie Lane Width     |Temp  MPC Capable  
                                         |      ECC Mode     
+-------------------------------------------------------------+
0    MTT S4000      |00000000:01:00.0    |0%    518MiB(49152MiB)
     Physical       |16x(16x)            |89C   YES          
                                         |      N/A          
---------------------------------------------------------------

---------------------------------------------------------------
Processes:
ID   PID       Process name                         GPU Memory
                                                         Usage
+-------------------------------------------------------------+
   No running processes found
---------------------------------------------------------------

GDS

SmartIO的GPUDirect Storage技术为了与mtfs.ko内核模块栈进行通信，需要可以访问字符设备。为了在Docker环境中启用GPUDirect Storage支持，新增了一个名为MTHREADS_GDS的环境变量，在使用docker command部署环境和在容器内测试GDS的时候来使能GPUDirect Storage支持

比如，如果你要创建container 使用GDS,你可以使用下面的命令

docker run -e MTHREADS_GDS=enable \
     -e MTHREADS_VISIBLE_DEVICES=all \
      [OPTIONS] IMAGE [COMMAND] [ARG...]

Note: MTHREADS_GDS must work with the MTHREADS_VISIBLE_DEVICES environment variable.

Dockerfiles

构建容器镜像时在 Dockerfile 添加环境变量，运行容器时就无需再指定它们。

ENV MTHREADS_VISIBLE_DEVICES all
ENV MTHREADS_DRIVER_CAPABILITIES compute,utility

Kubernetes(containerd, CRI-O)

和 Docker 一样，用户可以使用环境变量来控制器摩尔线程 Container Runtime 的行为。

下面是一些用法的示例：

使用 MTHREADS_VISIBLE_DEVICES 在所有 GPU 上启动一个支持 GPU 的容器：

pod.yaml:

 apiVersion: v1
 kind: Pod
 metadata:
 name: ubuntu
 labels:
     app: ubuntu
 spec:
 containers:
 - image: ubuntu
     command:
     - "sleep"
     - "604800"
     imagePullPolicy: Always
     name: ubuntu
     env:
     - name: MTHREADS_VISIBLE_DEVICES
     value: "all"
 restartPolicy: Always

 $ kubectl apply -f pod.yaml
 $ kubectl exec -it ubuntu bash
 root@ubuntu:/# mthreads-gmi
Thu Jul 10 11:52:47 2025
---------------------------------------------------------------
    mthreads-gmi:2.0.0           Driver Version:3.0.0
---------------------------------------------------------------
ID   Name           |PCIe                |%GPU  Mem          
     Device Type    |Pcie Lane Width     |Temp  MPC Capable  
                                         |      ECC Mode     
+-------------------------------------------------------------+
0    MTT S4000      |00000000:01:00.0    |0%    518MiB(49152MiB)
     Physical       |16x(16x)            |89C   YES          
                                         |      N/A          
---------------------------------------------------------------

---------------------------------------------------------------
Processes:
ID   PID       Process name                         GPU Memory
                                                         Usage
+-------------------------------------------------------------+
   No running processes found
---------------------------------------------------------------

使用 MTHREADS_VISIBLE_DEVICES 在特定的 GPU 上启动一个支持 GPU 的容器，并指定 compute 和 utility 能力，允许使用 MUSA 和 MTML：

pod.yaml:

 apiVersion: v1
 kind: Pod
 metadata:
 name: ubuntu
 labels:
     app: ubuntu
 spec:
 containers:
 - image: ubuntu
     command:
     - "sleep"
     - "604800"
     imagePullPolicy: Always
     name: ubuntu
     env:
     - name: MTHREADS_VISIBLE_DEVICES
     value: "1"
     - name: MTHREADS_DRIVER_CAPABILITIES
     value: "compute,utility"
 restartPolicy: Always

$ kubectl apply -f pod.yaml
$ kubectl exec -it ubuntu bash
 root@ubuntu:/# mthreads-gmi
Thu Jul 10 11:52:47 2025
---------------------------------------------------------------
    mthreads-gmi:2.0.0           Driver Version:3.0.0
---------------------------------------------------------------
ID   Name           |PCIe                |%GPU  Mem          
     Device Type    |Pcie Lane Width     |Temp  MPC Capable  
                                         |      ECC Mode     
+-------------------------------------------------------------+
0    MTT S4000      |00000000:01:00.0    |0%    518MiB(49152MiB)
     Physical       |16x(16x)            |89C   YES          
                                         |      N/A          
---------------------------------------------------------------

---------------------------------------------------------------
Processes:
ID   PID       Process name                         GPU Memory
                                                         Usage
+-------------------------------------------------------------+
   No running processes found
---------------------------------------------------------------

MT GPU Operator 使用指南

基本操作和维护

检查组件

检查 ClusterPolicy 的状态是否就绪：

$ kubectl get clusterpolicy mt-gpu-cluster-policy -o yaml
...
status:
  namespace: mt-gpu-operator
  state: ready

如果 status 没有出现在 ClusterPolicy 中，这意味着一些组件还没有准备好。你应该检查 mt-controller-manager Pod 日志：

$ kubectl -n mt-gpu-operator logs  mt-controller-manager-7bbfbfc44c-7f7x6 --tail=40
I0823 08:21:15.048441       1 object_controls.go:1111] controllers/ClusterPolicy "msg"="DEBUG: DaemonSet"  "LabelSelector"="app=node-problem-detector"
I0823 08:21:15.048509       1 object_controls.go:1117] controllers/ClusterPolicy "msg"="DEBUG: DaemonSet"  "NumberOfDaemonSets"=1
I0823 08:21:15.048530       1 object_controls.go:1123] controllers/ClusterPolicy "msg"="DEBUG: DaemonSet"  "NumberUnavailable"=0
I0823 08:21:15.048561       1 clusterpolicy_controller.go:212] controllers/ClusterPolicy "msg"="INFO: ClusterPolicy step completed"  "state:"="state-node-problem-detector" "status"="ready"
I0823 08:21:15.053998       1 object_controls.go:82] controllers/ClusterPolicy "msg"="Found Resource, skipping update" "Namespace"="mt-gpu-operator" "ServiceAccount"="mt-aiops"
I0823 08:21:15.066734       1 object_controls.go:207] controllers/ClusterPolicy "msg"="Found Resource, updating..." "ClusterRole"="mt-aiops" "Namespace"="mt-gpu-operator"
I0823 08:21:15.092702       1 object_controls.go:251] controllers/ClusterPolicy "msg"="Found Resource, updating..." "ClusterRoleBinding"="mt-aiops" "Namespace"="mt-gpu-operator"
I0823 08:21:15.105337       1 object_controls.go:1573] controllers/ClusterPolicy "msg"="Deployment identical, skipping update" "Deployment"="mt-aiops" "Namespace"="mt-gpu-operator" "name"="mt-aiops"
I0823 08:21:15.105373       1 object_controls.go:1486] controllers/ClusterPolicy "msg"="DEBUG: Deployment"  "LabelSelector"="app=mt-aiops"
I0823 08:21:15.105451       1 object_controls.go:1492] controllers/ClusterPolicy "msg"="DEBUG: Deployment"  "NumberOfDeployment"=1
I0823 08:21:15.105482       1 object_controls.go:1498] controllers/ClusterPolicy "msg"="DEBUG: Deployment"  "NumberUnavailable"=0
I0823 08:21:15.105504       1 object_controls.go:1461] controllers/ClusterPolicy "msg"="DEBUG: Pod"  "LabelSelector"="app=mt-aiops"
I0823 08:21:15.105611       1 object_controls.go:1467] controllers/ClusterPolicy "msg"="DEBUG: Pod"  "NumberOfPods"=1
I0823 08:21:15.105644       1 object_controls.go:1478] controllers/ClusterPolicy "msg"="DEBUG: Pod"  "=="="Running" "Phase"="Running"
I0823 08:21:15.105670       1 clusterpolicy_controller.go:212] controllers/ClusterPolicy "msg"="INFO: ClusterPolicy step completed"  "state:"="state-aiops" "status"="ready"
I0823 08:21:15.109247       1 object_controls.go:82] controllers/ClusterPolicy "msg"="Found Resource, skipping update" "Namespace"="mt-gpu-operator" "ServiceAccount"="mt-operator-validator"
I0823 08:21:15.110138       1 object_controls.go:1200] controllers/ClusterPolicy "msg"="DaemonSet identical, skipping update" "DaemonSet"="mt-operator-validator" "Namespace"="mt-gpu-operator" "name"="mt-operator-validator"
I0823 08:21:15.110163       1 object_controls.go:1111] controllers/ClusterPolicy "msg"="DEBUG: DaemonSet"  "LabelSelector"="app=mt-operator-validator"
I0823 08:21:15.110228       1 object_controls.go:1117] controllers/ClusterPolicy "msg"="DEBUG: DaemonSet"  "NumberOfDaemonSets"=1
I0823 08:21:15.110248       1 object_controls.go:1123] controllers/ClusterPolicy "msg"="DEBUG: DaemonSet"  "NumberUnavailable"=1
I0823 08:21:15.110279       1 clusterpolicy_controller.go:212] controllers/ClusterPolicy "msg"="INFO: ClusterPolicy step completed"  "state:"="state-operator-validation" "status"="notReady"
I0823 08:21:15.110909       1 object_controls.go:1200] controllers/ClusterPolicy "msg"="DaemonSet identical, skipping update" "DaemonSet"="mt-gpu-feature-discovery" "Namespace"="mt-gpu-operator" "name"="mt-gpu-feature-discovery"
I0823 08:21:15.110928       1 object_controls.go:1111] controllers/ClusterPolicy "msg"="DEBUG: DaemonSet"  "LabelSelector"="app=mt-gpu-feature-discovery"
I0823 08:21:15.110969       1 object_controls.go:1117] controllers/ClusterPolicy "msg"="DEBUG: DaemonSet"  "NumberOfDaemonSets"=1
I0823 08:21:15.110985       1 object_controls.go:1123] controllers/ClusterPolicy "msg"="DEBUG: DaemonSet"  "NumberUnavailable"=1
I0823 08:21:15.111007       1 clusterpolicy_controller.go:212] controllers/ClusterPolicy "msg"="INFO: ClusterPolicy step completed"  "state:"="state-gpu-feature-discovery" "status"="notReady"
I0823 08:21:15.115416       1 object_controls.go:82] controllers/ClusterPolicy "msg"="Found Resource, skipping update" "Namespace"="mt-gpu-operator" "ServiceAccount"="mt-device-plugin"
I0823 08:21:15.129355       1 object_controls.go:207] controllers/ClusterPolicy "msg"="Found Resource, updating..." "ClusterRole"="mt-device-plugin" "Namespace"="mt-gpu-operator"
I0823 08:21:15.154087       1 object_controls.go:251] controllers/ClusterPolicy "msg"="Found Resource, updating..." "ClusterRoleBinding"="mt-device-plugin" "Namespace"="mt-gpu-operator"
I0823 08:21:15.167282       1 object_controls.go:1200] controllers/ClusterPolicy "msg"="DaemonSet identical, skipping update" "DaemonSet"="mt-device-plugin" "Namespace"="mt-gpu-operator" "name"="mt-device-plugin"
I0823 08:21:15.167309       1 object_controls.go:1111] controllers/ClusterPolicy "msg"="DEBUG: DaemonSet"  "LabelSelector"="app=mt-device-plugin"
I0823 08:21:15.167363       1 object_controls.go:1117] controllers/ClusterPolicy "msg"="DEBUG: DaemonSet"  "NumberOfDaemonSets"=1
I0823 08:21:15.167383       1 object_controls.go:1123] controllers/ClusterPolicy "msg"="DEBUG: DaemonSet"  "NumberUnavailable"=1
I0823 08:21:15.167408       1 clusterpolicy_controller.go:212] controllers/ClusterPolicy "msg"="INFO: ClusterPolicy step completed"  "state:"="state-device-plugin" "status"="notReady"
I0823 08:21:15.167970       1 object_controls.go:1200] controllers/ClusterPolicy "msg"="DaemonSet identical, skipping update" "DaemonSet"="mt-gpu-exporter" "Namespace"="mt-gpu-operator" "name"="mt-gpu-exporter"
I0823 08:21:15.167991       1 object_controls.go:1111] controllers/ClusterPolicy "msg"="DEBUG: DaemonSet"  "LabelSelector"="app=mt-gpu-exporter"
I0823 08:21:15.168034       1 object_controls.go:1117] controllers/ClusterPolicy "msg"="DEBUG: DaemonSet"  "NumberOfDaemonSets"=1
I0823 08:21:15.168052       1 object_controls.go:1123] controllers/ClusterPolicy "msg"="DEBUG: DaemonSet"  "NumberUnavailable"=1
I0823 08:21:15.168075       1 clusterpolicy_controller.go:212] controllers/ClusterPolicy "msg"="INFO: ClusterPolicy step completed"  "state:"="state-gpu-exporter" "status"="notReady"
I0823 08:21:15.168101       1 clusterpolicy_controller.go:226] controllers/ClusterPolicy "msg"="ClusterPolicy isn't ready"  "states not ready"=["state-container-toolkit","state-operator-validation","state-gpu-feature-discovery","state-device-plugin","state-gpu-exporter"]
$ kubectl logs mt-controller-manager-7bbfbfc44c-7f7x6 --tail=20
I0823 08:21:42.445557       1 clusterpolicy_controller.go:212] controllers/ClusterPolicy "msg"="INFO: ClusterPolicy step completed"  "state:"="state-operator-validation" "status"="notReady"
I0823 08:21:42.446278       1 object_controls.go:1200] controllers/ClusterPolicy "msg"="DaemonSet identical, skipping update" "DaemonSet"="mt-gpu-feature-discovery" "Namespace"="mt-gpu-operator" "name"="mt-gpu-feature-discovery"
I0823 08:21:42.446300       1 object_controls.go:1111] controllers/ClusterPolicy "msg"="DEBUG: DaemonSet"  "LabelSelector"="app=mt-gpu-feature-discovery"
I0823 08:21:42.446349       1 object_controls.go:1117] controllers/ClusterPolicy "msg"="DEBUG: DaemonSet"  "NumberOfDaemonSets"=1
I0823 08:21:42.446368       1 object_controls.go:1123] controllers/ClusterPolicy "msg"="DEBUG: DaemonSet"  "NumberUnavailable"=1
I0823 08:21:42.446398       1 clusterpolicy_controller.go:212] controllers/ClusterPolicy "msg"="INFO: ClusterPolicy step completed"  "state:"="state-gpu-feature-discovery" "status"="notReady"
I0823 08:21:42.450740       1 object_controls.go:82] controllers/ClusterPolicy "msg"="Found Resource, skipping update" "Namespace"="mt-gpu-operator" "ServiceAccount"="mt-device-plugin"
I0823 08:21:42.462734       1 object_controls.go:207] controllers/ClusterPolicy "msg"="Found Resource, updating..." "ClusterRole"="mt-device-plugin" "Namespace"="mt-gpu-operator"
I0823 08:21:42.487206       1 object_controls.go:251] controllers/ClusterPolicy "msg"="Found Resource, updating..." "ClusterRoleBinding"="mt-device-plugin" "Namespace"="mt-gpu-operator"
I0823 08:21:42.499236       1 object_controls.go:1200] controllers/ClusterPolicy "msg"="DaemonSet identical, skipping update" "DaemonSet"="mt-device-plugin" "Namespace"="mt-gpu-operator" "name"="mt-device-plugin"
I0823 08:21:42.499264       1 object_controls.go:1111] controllers/ClusterPolicy "msg"="DEBUG: DaemonSet"  "LabelSelector"="app=mt-device-plugin"
I0823 08:21:42.499325       1 object_controls.go:1117] controllers/ClusterPolicy "msg"="DEBUG: DaemonSet"  "NumberOfDaemonSets"=1
I0823 08:21:42.499346       1 object_controls.go:1123] controllers/ClusterPolicy "msg"="DEBUG: DaemonSet"  "NumberUnavailable"=1
I0823 08:21:42.499371       1 clusterpolicy_controller.go:212] controllers/ClusterPolicy "msg"="INFO: ClusterPolicy step completed"  "state:"="state-device-plugin" "status"="notReady"
I0823 08:21:42.499931       1 object_controls.go:1200] controllers/ClusterPolicy "msg"="DaemonSet identical, skipping update" "DaemonSet"="mt-gpu-exporter" "Namespace"="mt-gpu-operator" "name"="mt-gpu-exporter"
I0823 08:21:42.499952       1 object_controls.go:1111] controllers/ClusterPolicy "msg"="DEBUG: DaemonSet"  "LabelSelector"="app=mt-gpu-exporter"
I0823 08:21:42.499998       1 object_controls.go:1117] controllers/ClusterPolicy "msg"="DEBUG: DaemonSet"  "NumberOfDaemonSets"=1
I0823 08:21:42.500017       1 object_controls.go:1123] controllers/ClusterPolicy "msg"="DEBUG: DaemonSet"  "NumberUnavailable"=1
I0823 08:21:42.500046       1 clusterpolicy_controller.go:212] controllers/ClusterPolicy "msg"="INFO: ClusterPolicy step completed"  "state:"="state-gpu-exporter" "status"="notReady"
I0823 08:21:42.500071       1 clusterpolicy_controller.go:226] controllers/ClusterPolicy "msg"="ClusterPolicy isn't ready"  "states not ready"=["state-container-toolkit","state-operator-validation","state-gpu-feature-discovery","state-device-plugin","state-gpu-exporter"]

日志会告诉你哪些组件还没有准备好。从上面的输出中我们可以看到，以下组件状态是有问题的：container-toolkit、operator-validator、gpu-feature-discovery、device-plugin 和 gpu-exporter。你可以向开发人员报告以下结果和有问题的组件状态：

$ kubectl -n mt-gpu-operator get po -o wide
NAME                                     READY   STATUS       RESTARTS      AGE
kured-f94df                              1/1     Running      0             62s
kured-k442c                              1/1     Running      0             62s
kured-lgwt4                              1/1     Running      0             62s
kured-xhd8d                              1/1     Running      0             62s
mt-aiops-c988dc879-4pnst                 1/1     Running      0             62s
mt-container-toolkit-daemonset-24w6f     1/1     Running      0             62s
mt-container-toolkit-daemonset-rtvr5     0/1     Init:Error   3 (43s ago)   62s
mt-controller-manager-7bbfbfc44c-zl9tr   2/2     Running      0             106s
mt-device-plugin-56hxn                   0/1     Init:0/1     0             60s
mt-device-plugin-nqnnn                   1/1     Running      0             61s
mt-driver-toolkit-89lsm                  1/1     Running      0             62s
mt-driver-toolkit-nwv92                  1/1     Running      0             62s
mt-gpu-exporter-fwqww                    1/1     Running      0             60s
mt-gpu-exporter-jf26z                    0/1     Init:0/1     0             60s
mt-gpu-feature-discovery-djh8g           0/1     Init:0/1     0             61s
mt-gpu-feature-discovery-p2csq           1/1     Running      0             61s
mt-gpu-webhook-5b58696885-2fqpg          1/1     Running      0             62s
mt-gpu-webhook-gen-secret-d9pl2          0/1     Completed    0             62s
mt-gpushare-scheduler-578d5595f9-h59rv   1/1     Running      0             62s
mt-operator-validator-mnlpt              0/1     Init:0/2     0             61s
mt-operator-validator-zzkgh              1/1     Running      0             61s
nfd-master-847694d746-5xw54              1/1     Running      0             63s
nfd-worker-bjg8v                         1/1     Running      0             62s
nfd-worker-g8pdl                         1/1     Running      0             62s
nfd-worker-vm2zx                         1/1     Running      0             62s
nfd-worker-wl5j4                         1/1     Running      0             62s
node-problem-detector-hq4lr              1/1     Running      0             62s
node-problem-detector-hztjz              1/1     Running      0             61s
node-problem-detector-j77rg              1/1     Running      0             61s
node-problem-detector-q7fmx              1/1     Running      0             62s

驱动安装基本运维

查询基本状态

如果驱动安装/升级失败，你可以通过以下命令查看驱动当前安装的状态：

$ kubectl get nodeconfig
node-1      InstallDone
node-2      InstallDone

当驱动安装失败后，可以查看驱动升级的相关事件：

$ kubectl get events --sort-by='.lastTimestamp' | grep DriverUpgrade
...
5m28s       Normal    DriverUpgrade              nodeconfig/node-1                             (combined from similar events): Successfully updated nodeconfig state label to ValidateNode
5m19s       Warning   DriverUpgrade              nodeconfig/node-1                               Failed to update nodeconfig state label to UpgradeFailed err: timed out waiting for the condition
4m28s       Normal    DriverUpgrade              nodeconfig/node-1                                  Successfully updated nodeconfig state label to WaitForJobsRequired
79s         Normal    DriverUpgrade              nodeconfig/node-1                                 Successfully updated nodeconfig state label to InstallDone
29s         Normal    DriverUpgrade              nodeconfig/node-2                                                  Successfully updated nodeconfig state label to InstallDone
...

你也可以单独查看某一个节点上驱动安装事件：

$ kubectl describe nodeconfig <node-name>
...
Events:
  Type     Reason         Age   From             Message
  ----     ------         ----  ----             -------
  Warning  DriverUpgrade  12s   mt-gpu-operator  2024-01-19 03:29:06: failed reason: timed out waiting for the condition
  Normal   DriverUpgrade  12s   mt-gpu-operator  2024-01-19 03:29:06: disable mt-gpu-operator deploy label successfully, node: yuzhou-system-product-name
  Normal   DriverUpgrade  12s   mt-gpu-operator  2024-01-19 03:29:06: Successfully drained the node
  Normal   DriverInstall  12s   mt-gpu-operator  2024-01-19 03:29:06: Success to clear gpu

升级失败

用户可以通过查看 mt-controller-manager 日志来定位问题：

km logs mt-controller-manager-xxx-xxx | grep controllers.NodeConfig

在解决升级失败问题后，可以通过将NodeConfig标签设置为升级所需的状态来继续升级过程。

$ kubectl label node <node-name>  mthreads.com/gpu-driver-upgrade-state=UpgradeRequired --overwrite

节点卡在 RebootNodeRequired 状态

如果节点长时间卡在 RebootNodeRequired 状态：

$ kubectl get nodeconfig
node-1      RebootNodeRequired

可能是由 Kured 导致的问题。在重启节点之前，Kured 会尝试驱逐节点上的负载，但这一步骤可能会失败。用户可以通过查看节点上的注解和标签来确定 Kured 是否卡在驱逐阶段：

$ kubectl describe node <node-name> | grep reboot
                    mthreads.com/node-need-reboot=true
                    weave.works/kured-most-recent-reboot-needed: 2024-01-18T04:58:47Z
                    weave.works/kured-reboot-in-progress: 2024-01-18T04:58:47Z

这些标签和注解是由 Kured 创建的：

mthreads.com/node-need-reboot=true：表示节点需要重启。
weave.works/kured-most-recent-reboot-needed: 2024-01-18T04:58:47Z：表示上一次由 Kured 触发节点重启的时间。
weave.works/kured-reboot-in-progress: 2024-01-18T04:58:47Z：表示当前 Kured 正在处理节点，可能正在驱逐节点上的负载。

若希望在 Kured 驱逐失败时强制重启节点，在 ClusterPolicy 中添加 Kured 强制重启的参数：

  kured:
    enabled: true
    args:
    ...
    - --force-reboot=true

MT Container Toolkit 使用指南​

Docker​

GPU 枚举​

驱动能力​

sGPU​

GDS​

Dockerfiles​

Kubernetes(containerd, CRI-O)​

MT GPU Operator 使用指南​

基本操作和维护​

检查组件​

驱动安装基本运维​

查询基本状态​

升级失败​

节点卡在 RebootNodeRequired 状态​

这篇文章对您有帮助吗？