GPU autoscaling
GPU nodes that follow demand, in both directions
The GPU optimizer adds GPU nodes when pods wait for them, uses spot GPUs with an on-demand fallback, packs GPU pods so empty nodes can leave, and scales idle GPU workloads to zero. It never goes past a policy's node, cost or GPU-type limits, and every action can be reviewed and reverted.
GPU autoscaling workflow
Who does what
GPU nodes that follow demand
Follow the inputs, decision and output for this workflow.
- KubeOn, automated
- Your decision
- Your systems and people
- Only path to a change or purchase
- Step 1, Watch, by GPU optimizer:
Pending GPU pods, GPU requests per node and DCGM utilization, every minute.
- Step 2, Add, by GPU optimizer:
Add a GPU node of an allowed type, spot first, on-demand when spot is unavailable or reclaimed.
- Step 3, Pack + remove, by GPU optimizer:
When pods fit on fewer nodes, drain the emptiest within PodDisruptionBudgets and remove it.
- Step 4, Review, by Platform team:
Check the actions, the cost against the policy's limit, and revert or tighten the policy.
The optimizer never goes past a policy's minimum and maximum nodes, monthly cost limit or allowed GPU types.
Read the workflowWho does what
GPU nodes that follow demand
Follow the inputs, decision and output for this workflow.
- 01
Watch
Pending GPU pods, GPU requests per node and DCGM utilization, every minute.
GPU optimizer
- 02
Add
Add a GPU node of an allowed type, spot first, on-demand when spot is unavailable or reclaimed.
GPU optimizer
- 03
Pack + remove
When pods fit on fewer nodes, drain the emptiest within PodDisruptionBudgets and remove it.
GPU optimizer
- 04
Review
Check the actions, the cost against the policy's limit, and revert or tighten the policy.
Platform team
The optimizer never goes beyond a policy's minimum and maximum nodes, monthly cost limit or allowed GPU types.
Read the workflowThe problem
GPU pools sized for the busiest hour, all day
Fixed pools
GPU node pools are sized once for the peak and left running through nights and weekends.
Spread-out pods
GPU pods land on whichever node is free, so no node ever empties enough to be removed.
Spot feels risky
Spot GPUs cost less, but a reclaimed node without a fallback means failed requests.
Idle notebooks
An interactive session holds a GPU while nobody is using it, often overnight.
Capacity on demand
GPU nodes added when GPU pods wait
When GPU pods are pending, the optimizer adds the cheapest allowed GPU node that fits them, through Karpenter or your cloud's node groups. It checks the policy's maximum nodes and monthly cost first, and stops at the limit instead of passing it.
- Allowed GPU types and sizes per policy
- Minimum and maximum nodes, monthly cost cap
- Pending pods that would breach a limit are reported, not hidden
apiVersion: gpu.kubeon.io/v1kind: GpuPolicymetadata: name: ml-inference-autoscalingspec: kind: Autoscaling mode: Automatic # Recommend | Approve | Automatic nodePools: [ml-inference] limits: minNodes: 1 maxNodes: 6 maxMonthlyCost: 40000 # USD at billed rates capacity: order: [spot, on-demand] # spot first, on-demand when spot is gone packing: enabled: true window: "Mon-Fri 19:00-23:00 UTC" respectPodDisruptionBudgets: trueSpot GPUs
Spot first, on-demand when spot goes away
Policies can prefer spot GPUs and fall back to on-demand when spot capacity is unavailable or reclaimed. On an interruption notice, two minutes on AWS and 30 seconds on Azure and Google Cloud, the optimizer cordons the node, starts a replacement and lets pods drain in time.
- Capacity order per policy: spot, then on-demand
- Replacement started on the interruption notice
- Spot only for workloads that tolerate restarts
Packing and consolidation
Pods packed so empty GPU nodes can leave
GPU pods are placed on the fullest node that fits them, aware of MIG slices and time-sliced GPUs. In the policy's window, when the pods would fit on fewer nodes, the optimizer drains the emptiest node through the Eviction API, within PodDisruptionBudgets, and removes it.
- GPU-aware placement, MIG and time-slicing included
- Consolidation only inside the change window
- Each removal priced at the node's billed rate
Scale to zero
Idle GPU workloads scaled to zero, nodes removed
In the namespaces a policy lists, a GPU workload whose GPU utilization stays under 2% for the idle period is scaled to zero replicas. Its volumes and configuration stay. When its node is empty, the node is removed, and the workload comes back on schedule or when someone scales it up.
- Only namespaces named in the policy
- Idle measured from DCGM, not guessed
- Wake on a schedule or on demand
apiVersion: gpu.kubeon.io/v1kind: GpuPolicymetadata: name: notebooks-scale-to-zerospec: kind: ScaleToZero mode: Automatic namespaces: [ml-notebooks] # only namespaces you list idleAfter: 30m # GPU utilization under 2% (DCGM) wake: schedule: "Mon-Fri 07:00 UTC" removeEmptyNodes: trueAutoscaling rules
What the optimizer does, and the limit on each action
| Action | When | Limit |
|---|---|---|
| Add a GPU node | GPU pods pending that an allowed GPU type can hold | Maximum nodes and monthly cost per policy |
| Replace a spot node | Interruption notice, or spot unavailable when scaling up | On-demand only if the policy allows it |
| Pack and remove a node | Pods fit on fewer nodes | Change window and PodDisruptionBudgets |
| Scale a workload to zero | GPU utilization under 2% for the idle period | Only the namespaces the policy lists |
| Smaller GPU type | Peak GPU memory plus 30% fits a smaller GPU class | Always a proposal: latency needs a benchmark |
Commitments still apply: removing a node covered by a Savings Plan or reservation moves the commitment to other usage rather than cutting the bill, so KubeOn prices removals at your net amortized rate.
Capabilities
Autoscaling with limits you set
Scale up
GPU nodes added for pending GPU pods.
Spot with fallback
Spot first, on-demand when spot is gone.
GPU-aware packing
Fewer, fuller GPU nodes, MIG included.
Scale to zero
Idle GPU workloads off, empty nodes removed.
Hard limits
Nodes, monthly cost and GPU types per policy.
Revert
Undo a removal or a scale-to-zero from the log.
FAQ
GPU autoscaling, answered
No. The optimizer drives GPU capacity through Karpenter NodePools or your cloud's node groups, and leaves CPU node pools to the autoscaler you already run.
Related
Evaluate GPU autoscaling with your team.
Bring your platform and FinOps priorities. We will walk through the data, controls and evidence for your evaluation.