What about training jobs that need many GPUs at once?

Pools that run distributed training are sized from the peak including those jobs. If a large job runs less often than every 14 days, reject the proposal or exclude the pool.

Does capacity sizing account for reservations?

Savings are priced at the billed, net amortized cost, so covered hours are priced at the committed rate. Removing a node covered by a commitment moves the commitment to other usage rather than cutting the bill, so check coverage before approving.

GPU capacity

GPU node pools sized from what pods actually request

KubeOn sizes each GPU node pool from its peak hourly GPU requests, flags workloads whose GPU memory fits a smaller GPU type, and schedules development GPUs off outside working hours. Each finding is priced at the pool's billed rate and becomes a proposal.

Book a demo Capacity rules

GPU capacity

Node pools and GPU types sized from requests and GPU memory

Scope	Finding	Evidence	Savings
prod-use1-a / ml-inference	Scale 3 nodes to 2	Peak GPU requests over 14 days: 7 of 12. Two nodes hold 8.	$8,433.33/mo
prod-use1-a / rerank-svc	Fits a smaller GPU type	Peak GPU memory 14 GB of 48 GB. A 24 GB class GPU fits with headroom.	Review
dev-use1 / ml-dev	Off outside 07:00 to 20:00 on weekdays	103 idle hours a week on a development GPU node.	$1,042.26/mo

Each finding becomes a proposal. Nothing changes until your team approves it and your pipeline applies it.

Who does what

Size GPU node pools to real demand

Follow the inputs, decision and output for this workflow.

01
Collect
Hourly GPU requests per node pool, and GPU memory peaks per workload.
Agent
02
Size
Fewer GPU nodes where peak requests leave whole nodes empty; smaller GPU types where models fit.
KubeOn hub
03
Approve
Check latency targets and scheduled peaks the history window might not contain.
Platform team
04
Apply
Change node pool size, GPU type or schedule through your infrastructure code.
Your pipeline

KubeOn reads and recommends. Your team and pipeline change the cluster.

Read the workflow

Node pool size

Fewer GPU nodes where whole nodes sit empty at peak

For each GPU node pool, KubeOn takes the peak of hourly GPU requests over 14 days and counts how many nodes would hold it. When whole nodes would be left over even at the busiest hour, it proposes a smaller minimum and desired size, and keeps the maximum so the pool can scale back up.

Peak hourly GPU requests over 14 days
Whole nodes only: a GPU node is never half removed
Terraform, eksctl or Karpenter changes written out

Node group change

# prod-use1-a / ml-inference: peak GPU requests 7 of 12 over 14 daysresource "aws_eks_node_group" "ml_inference" {  cluster_name   = "prod-use1-a"  instance_types = ["g6e.12xlarge"] # 4 x NVIDIA L40S  scaling_config {    min_size     = 2 # was 3    desired_size = 2 # was 3    max_size     = 3 # room to scale back up  }}

GPU capacity

Node pools and GPU types sized from requests and GPU memory

Scope	Finding	Evidence	Savings
prod-use1-a / ml-inference	Scale 3 nodes to 2	Peak GPU requests over 14 days: 7 of 12. Two nodes hold 8.	$8,433.33/mo
prod-use1-a / rerank-svc	Fits a smaller GPU type	Peak GPU memory 14 GB of 48 GB. A 24 GB class GPU fits with headroom.	Review
dev-use1 / ml-dev	Off outside 07:00 to 20:00 on weekdays	103 idle hours a week on a development GPU node.	$1,042.26/mo

Each finding becomes a proposal. Nothing changes until your team approves it and your pipeline applies it.

GPU type

Models that fit a smaller GPU

A model whose peak GPU memory is 14 GB does not need a 48 GB card. KubeOn compares each inference workload's peak GPU memory with the memory of the GPU it runs on and flags those that fit a smaller GPU class with headroom. Because speed differs between GPU types, these are review findings, not counted savings, until your team benchmarks them.

Peak GPU memory with 30% headroom
Same vendor and driver family
Latency is for your team to confirm

GPU type fit

prod-use1-a / rerank-svc on NVIDIA L40S (48 GB)peak GPU memory, 14 days     14.0 GBwith 30% headroom            18.2 GBfits                         24 GB class (NVIDIA L4, A10G)status                       review: benchmark latency first

Off-hours

Development GPUs off outside working hours

GPU nodes in dev, test and sandbox clusters get an off-hours proposal: GPU pods scaled down outside 07:00 to 20:00 on weekdays, and the empty nodes removed by the autoscaler. The saving is estimated on the 103 hours a week they would be idle.

kube-downscaler annotations on the namespace
Node group minimum at 0 so empty GPU nodes leave
Non-production only

Off-hours schedule

# dev-use1 / ml-notebooks: GPU pods scale down outside working hours.# With the ml-dev node group at a minimum of 0, the empty GPU node is removed.apiVersion: v1kind: Namespacemetadata:  name: ml-notebooks  annotations:    downscaler/uptime: Mon-Fri 07:00-20:00 UTC

Capacity rules

Every GPU capacity finding, with its rule

Finding	Rule	Saving
Scale a GPU node pool	Peak hourly GPU requests over 14 days fit on fewer nodes, leaving at least one whole node	Billed cost of the nodes removed
Smaller GPU type	Peak GPU memory plus 30% fits a smaller GPU class	Review only, until benchmarked
GPU off-hours	GPU nodes in non-production clusters, idle outside 07:00 to 20:00 on weekdays	103 of 168 hours a week of the node's cost
Idle GPU nodes	GPU nodes with no GPU requested for 7 days	The node's full billed cost

14 days does not contain a quarterly training run or a seasonal peak. Reject proposals for pools whose busiest days fall outside the window.

Capabilities

GPU capacity planned from evidence

Pool sizing

Fewer GPU nodes where peak requests leave nodes empty.

GPU type fit

Peak GPU memory against the card's memory.

Off-hours

Development GPU nodes off at night and at weekends.

Idle GPU nodes

Nodes with no GPU requested for a week.

Stated windows

Every finding says which days it looked at.

Approved changes

Proposals with a risk level, an owner and a patch.

FAQ

GPU capacity, answered

No. KubeOn writes the change to your node group, NodePool or schedule. Your team approves it and your pipeline applies it.

Splits each GPU node's billed cost by the GPUs pods requested, reports GPUs nobody requested as idle GPU time, and shows how busy the allocated GPUs really are, from NVIDIA DCGM metrics.

Learn more

GPU sharing

Finds workloads whose GPU compute and memory fit in a fraction of a card, and proposes MIG partitions on supported GPUs or time-slicing for development, with the GPU Operator configuration and pod changes written out for review.

Learn more

Savings

Container requests are sized from p95 usage over 7 days, node groups from 14 days of peak requests. Orphaned volumes, abandoned workloads and off-hours schedules are priced per month and recomputed every 30 minutes.

Learn more

Evaluate GPU capacity with your team.

Bring your platform and FinOps priorities. We will walk through the data, controls and evidence for your evaluation.

Book a demo Read the workflow

GPU node pools sized from what pods actually request

GPU capacity workflow

Fewer GPU nodes where whole nodes sit empty at peak

Models that fit a smaller GPU

Development GPUs off outside working hours

Every GPU capacity finding, with its rule

GPU capacity planned from evidence

Pool sizing

GPU type fit

Off-hours

Idle GPU nodes

Stated windows

Approved changes

GPU capacity, answered

Does KubeOn scale our GPU nodes?

What about training jobs that need many GPUs at once?

Does capacity sizing account for reservations?

Evaluate GPU capacity with your team.