GPU cost and utilization
Every GPU-hour, with an owner and a utilization
KubeOn splits each GPU node's cost by the GPUs pods requested, hour by hour, reports the GPUs nobody requested as idle GPU time, and shows how busy the allocated GPUs really were. Cloud GPUs at your billed rate, owned GPUs at your rate card.
GPU cost and utilization workflow
Who does what
From a GPU node's bill to a team's GPU cost
Follow the inputs, decision and output for this workflow.
- KubeOn, automated
- Your decision
- Your systems and people
- Step 1, Price the node, by Billing or rate card:
Billed, net amortized cost of each GPU node, or a GPU-hour rate for servers you own.
- Step 2, Measure, by Agent + Prometheus:
GPU capacity and GPU requests from the Kubernetes API; utilization and memory from DCGM metrics.
- Step 3, Allocate, by KubeOn hub:
Split node cost by GPU requests per hour, and report GPUs no pod requested as idle GPU time.
- Step 4, Review, by Your teams:
Each team sees its GPU cost and how busy its GPUs were, next to the rest of its spend.
KubeOn reads and recommends. Your team and pipeline change the cluster.
Read the workflowWho does what
From a GPU node's bill to a team's GPU cost
Follow the inputs, decision and output for this workflow.
- 01
Price the node
Billed, net amortized cost of each GPU node, or a GPU-hour rate for servers you own.
Billing or rate card
- 02
Measure
GPU capacity and GPU requests from the Kubernetes API; utilization and memory from DCGM metrics.
Agent + Prometheus
- 03
Allocate
Split node cost by GPU requests per hour, and report GPUs no pod requested as idle GPU time.
KubeOn hub
- 04
Review
Each team sees its GPU cost and how busy its GPUs were, next to the rest of its spend.
Your teams
KubeOn reads and recommends. Your team and pipeline change the cluster.
Read the workflowThe problem
The most expensive nodes have the least visibility
Shared GPU nodes, no owner
Several teams' pods run on the same GPU nodes, so the bill shows one large line nobody owns.
Allocated is not used
A pod holding a GPU looks fully utilized to the scheduler, even when its kernels run a few percent of the time.
Idle cards hide in totals
GPUs no pod requested disappear into a cluster average, at several dollars an hour each.
Owned GPUs look free
On-premises GPU servers have no hourly bill, so their idle hours never reach a budget.
GPU cost allocation
GPU node cost split by the GPUs each pod requested
Kubernetes hands out GPUs as whole devices in a container's limits. KubeOn reads each node's GPU capacity and every pod's GPU request from the API each hour, and splits the node's billed, net amortized cost by the GPUs held. Teams see GPU cost next to the rest of their spend, by namespace, label and cost center.
- Net amortized node cost, commitments and discounts applied
- Hourly GPU requests per pod, including MIG slices
- Same owners and reports as CPU and memory cost
Idle GPU time
GPUs no pod requested, priced on their own
GPU capacity that no pod held is idle GPU time, priced at the node's rate and kept in its own column. It is a scheduling problem, with a different fix from low utilization: fewer nodes, a smaller pool, or a schedule. Each GPU node pool shows its allocated and idle GPUs and what the idle ones cost.
- Idle GPU-hours per node pool, per day
- Kept out of team totals unless your idle policy spreads it
- Feeds GPU capacity proposals directly
prod-use1-a / ml-inference g6e.12xlarge x 3, 4 x L40S eachGPU capacity 12 GPUs x 730 h = 8,760 GPU-hoursGPU requests (pods) 7 GPUs x 730 h = 5,110 GPU-hoursidle GPU time 3,650 GPU-hoursidle share of the pool's cost 3,650 / 8,760 = 41.7%Real utilization
How busy the allocated GPUs were, from DCGM
With the NVIDIA DCGM exporter scraped by Prometheus, KubeOn reads GPU utilization and memory per GPU and maps them to pods. GPU utilization counts the time any kernel was running, so it overstates how hard a GPU worked; where profiling metrics are on, SM activity is reported next to it.
- GPU utilization and memory used, per pod, per hour
- SM activity where profiling metrics are enabled
- Without DCGM, cost and idle GPU time still work from requests
# Prometheus scrape for the NVIDIA DCGM exporter (GPU Operator)# KubeOn reads these series per GPU, per podDCGM_FI_DEV_GPU_UTIL # % of time a kernel was runningDCGM_FI_DEV_FB_USED # GPU memory in use, MiBDCGM_FI_DEV_FB_FREE # GPU memory free, MiBDCGM_FI_PROF_SM_ACTIVE # SM activity, where profiling metrics are onSelf-hosted models
GPUs that serve your own models, next to hosted model spend
GPU nodes running your own models appear in KubeOn AI beside Azure OpenAI, Bedrock and Vertex AI spend. Compare what an inference service costs per month on your GPUs with what the same traffic would cost per token on a hosted model.
- Self-hosted GPU cost in the AI overview
- Shared owners, budgets and alerts
- Hosted comparison in cost per 1M tokens
Three numbers per GPU node
Allocated, idle and used are different problems
| Measure | What it tells you | Source | Typical fix |
|---|---|---|---|
| Allocated GPU-hours | How many GPUs pods held, and for how long | Pod GPU requests from the Kubernetes API | Owner and chargeback |
| Idle GPU-hours | GPU capacity no pod held | Node GPU capacity minus allocated GPUs | Fewer or smaller GPU nodes, schedules |
| Utilization of allocated GPUs | Whether the GPUs pods hold are doing work | DCGM exporter through Prometheus | Sharing, a smaller GPU type, or a hosted model |
GPU cost uses the same allocation method as the rest of KubeOn: billed, net amortized node cost, hourly requests, and idle and shared cost in their own columns.
Capabilities
GPU spend your teams can see and own
GPU allocation
Node cost split by GPU requests, per pod and per hour.
Idle GPU time
Unrequested GPUs priced at the node's rate.
DCGM utilization
GPU utilization, memory and SM activity per pod.
Team GPU cost
By namespace, label and cost center, effective-dated.
Billed rates
Commitments and discounts applied; rate cards on premises.
MIG aware
MIG slices allocated by the share of the GPU they hold.
FAQ
GPU cost and utilization, answered
Any NVIDIA GPU exposed through the NVIDIA device plugin or GPU Operator, in EKS, AKS, GKE or your own clusters. Utilization needs the DCGM exporter; cost and idle GPU time only need the Kubernetes API and your billing or rate card.
Related
Evaluate GPU cost and utilization with your team.
Bring your platform and FinOps priorities. We will walk through the data, controls and evidence for your evaluation.