Connect GPU metrics
Scrape the NVIDIA DCGM exporter with Prometheus so KubeOn can report GPU utilization and memory per pod.
On this page
GPU cost and idle GPU time need only the Kubernetes API and your bill. Utilization, sharing and scale to zero also need GPU metrics from the NVIDIA DCGM exporter, read through your Prometheus.
1. Run the DCGM exporter
The NVIDIA GPU Operator installs the DCGM exporter on every GPU node by default. Without the GPU Operator, install the dcgm-exporter Helm chart from NVIDIA as a DaemonSet on your GPU nodes. Leave pod mapping on, so each series carries the namespace and pod that hold the GPU.
2. Scrape it with Prometheus
# The GPU Operator installs the DCGM exporter on every GPU node.# Scrape it with the Prometheus Operator:apiVersion: monitoring.coreos.com/v1kind: ServiceMonitormetadata: name: nvidia-dcgm-exporter namespace: gpu-operatorspec: selector: matchLabels: app: nvidia-dcgm-exporter endpoints: - port: gpu-metrics interval: 30sWithout the Prometheus Operator, add a scrape job for the exporter's service on port 9400.
3. Point the agent at Prometheus
Set prometheus.url in the agent's values, as described in install the agent. The optimizer uses the same Prometheus, set in its own values.
4. Check the series
# Each series must carry the pod and namespace that hold the GPUcurl -s "$PROMETHEUS/api/v1/query" --data-urlencode 'query=count by (namespace, pod) (DCGM_FI_DEV_GPU_UTIL)'| Series | Used for |
|---|---|
DCGM_FI_DEV_GPU_UTIL | GPU utilization: the share of time any kernel was running. The idle signal for scale to zero |
DCGM_FI_DEV_FB_USED, DCGM_FI_DEV_FB_FREE | GPU memory in use and free. Decides MIG profiles, time-slicing replicas and GPU type fit |
DCGM_FI_PROF_SM_ACTIVE | Streaming-multiprocessor activity, where profiling metrics are enabled. Shown next to GPU utilization because it is the closer measure of how hard a GPU worked |
MIG slices