Idle GPUs: the most expensive capacity you don't use

GPUs are the fastest-growing line in most cloud bills. Across Datadog's customers, spend on GPU instances rose from 10% to 14% of EC2 compute costs in a single year, a 40% increase (Datadog, State of Cloud Costs 2024). Gartner forecast worldwide generative AI spending of $644 billion in 2025, up 76.4%, with about 80% of it going to hardware (Gartner, GenAI spending forecast (31 March 2025)).

Each GPU node costs many times what a general-purpose node does, so every idle hour on one costs as much as many idle hours elsewhere. And the published research on how GPUs are actually used in shared clusters is sobering.

What the research found

Microsoft's analysis of a two-month trace from a large multi-tenant GPU cluster used for deep learning training found that "the hardware utilization of GPUs in use is only around 52% on average" (Jeon et al., Analysis of Large-Scale Multi-Tenant GPU Clusters for DNN Training Workloads, USENIX ATC 2019). The causes were not lazy users: jobs waited for GPUs on the same machine, placement across servers slowed communication, and failed or killed jobs consumed GPU time without finishing.

Alibaba's study of a production machine-learning platform with over 6,000 GPUs found the gap is wider for inference-heavy work. A typical task used only a fraction of a GPU: the median was 0.042 GPUs of streaming-multiprocessor capacity, and over 75% of the click-through-rate prediction instances used less than 0.1 GPUs. On one group of machines, 74% of GPUs were allocated while average GPU utilization was 18% (Weng et al., MLaaS in the Wild: Workload Analysis and Scheduling in Large-Scale Heterogeneous GPU Clusters, USENIX NSDI 2022). The authors' conclusion was that allocating whole GPUs to tasks like these "would result in extremely low utilization", and they shared GPUs between tasks instead.

Why GPUs sit idle in Kubernetes

• Whole-device allocation. GPUs are requested in a container's limits, and requests must equal limits (Kubernetes, Schedule GPUs). Unless you enable sharing, the device plugin hands each GPU to one container, so a small inference service that needs a fraction of a card holds all of it.
• Provisioned for the peak. Inference replicas are sized for the busiest minute and then run all day, including nights and weekends.
• Notebooks and experiments. An interactive session holds a GPU while someone reads, thinks or goes to lunch.
• Waiting to start. Distributed training jobs need several GPUs at once, sometimes on the same machine; GPUs held while a job assembles do no work.
• Nobody sees the bill. GPU nodes are often shared by several teams, so the most expensive idle hours belong to nobody.

Measure three numbers per GPU node

Price each of them at the GPU node's real, billed rate. A node billed several dollars an hour turns a 30% idle rate into a number that gets a conversation started.

Options, from simplest to most involved

The right answer differs per workload, which is why the measurement comes first: a training job that runs at 90% utilization on 8 GPUs needs nothing, while an inference service holding one GPU at 6% is a sharing or hosting candidate. For the hosted-model comparison, see what a million tokens really costs.

GPUs in your own data center

On-premises GPU servers have no hourly bill, so their idle time is invisible. Give them a GPU-hour rate from the hardware's purchase price over its depreciation period, plus power, space and support, and the same three numbers appear. See pricing Kubernetes in your own data center.