FinOps
Where Kubernetes spend goes: what the public data says
Independent surveys and billing data agree: most of what you pay for in a Kubernetes cluster is capacity no workload uses. Where it goes, why Kubernetes makes it worse, and the order to fix it in.
KubeOn Engineering
October 16, 20266 min read
Cloud budgets are under pressure everywhere. In Flexera's 2025 survey of more than 750 technology leaders, 84% said managing cloud spend is their top cloud challenge; budgets were already 17% over their limits, and spend was expected to grow another 28% (Flexera, State of the Cloud 2025 (press release, 19 March 2025)). FinOps practitioners put workload optimization and waste reduction first, by a clear margin, in the FinOps Foundation's 2025 survey of 861 practitioners responsible for about $69 billion of cloud spend (FinOps Foundation, State of FinOps 2025).
For most of these organizations, the largest single place that waste hides is Kubernetes. Not because Kubernetes is wasteful by design, but because it puts a scheduling layer between the bill and the teams that cause it.
49%
of organizations said Kubernetes increased their cloud spend; 24% said it lowered it
Source: InfoQ, CNCF survey on Kubernetes overspend (March 2024)
70%
of respondents named overprovisioning as a cause of Kubernetes overspending
Source: InfoQ, CNCF survey on Kubernetes overspend (March 2024)
35%
of the average organization's EC2 compute spend now runs containers, up from 30%
Two kinds of idle
Datadog's State of Cloud Costs report splits wasted container spend into two parts (Datadog, State of Cloud Costs 2024). Cluster idle is capacity on nodes that no pod has requested: the cost of running more or bigger machines than the scheduled workloads need. Workload idle is capacity pods have requested but do not use: requests set higher than the workload's real consumption.
Where container cost went, across Datadog's customers
- 54% Cluster idle: nodes larger than the pods on them
- 29% Workload idle: requests larger than usage
- 17% Used by workloads (the remainder)
The distinction matters because each kind has a different owner and a different fix. Workload idle belongs to the team that set the requests, and the fix is a smaller request. Cluster idle belongs to whoever sizes the node groups, and the fix is fewer or smaller nodes. Shrink nodes first and pods with inflated requests stop fitting; shrink requests first and the freed capacity simply moves from workload idle into cluster idle until the nodes scale down.
Why Kubernetes makes this worse than plain virtual machines
- The scheduler reserves requests, not usage. A pod is placed on a node only if the node has room for its requests, whether or not the pod ever uses them (Kubernetes, Resource management for pods and containers). Over-sized requests take real capacity off the market.
- Requests are copied, not measured. A new service starts from the last service's manifest. Nobody revisits the numbers once the service is stable, because nothing breaks.
- Autoscalers scale on requests. A cluster autoscaler adds nodes when pods cannot be scheduled. Inflated requests trigger new nodes that sit mostly empty.
- Nobody owns a shared node. Forty teams run on the same node pool. The bill lists the node, not the namespaces, so the cost belongs to everyone and nobody acts on it.
- Resources outlive the workloads. In the same CNCF survey, 43% of respondents cited resources that were not shut down after use (InfoQ, CNCF survey on Kubernetes overspend (March 2024)).
What it looks like in one cluster
Take a production cluster billed $56,000 for the month. Pods request 60% of its CPU and memory, and use 25% of it. The bill shows one number; the split shows who can act on it.
| Part of the bill | Share of capacity | Monthly cost | Who can act |
|---|---|---|---|
| Used by workloads | 25% | $14,000 | Nobody: this is the work |
| Workload idle (requested, unused) | 35% | $19,600 | Each team, by lowering requests |
| Cluster idle (not requested) | 40% | $22,400 | The platform team, by shrinking node groups |
| Billed | 100% | $56,000 |
Two thirds of this cluster's bill is capacity nobody used, and almost none of it would show up in a report that only lists node cost per account.
Fix it in this order
- Allocate from the bill. Split each node's billed cost across the pods that ran on it, and keep idle, shared and unallocated cost in their own columns. See cost allocation that reconciles to the bill.
- Right-size requests. Size each container from its measured p95 usage with headroom, through the team's normal review. See rightsizing with p95, headroom and a floor.
- Then shrink nodes. Once requests come down, node groups can get smaller or move to a better-fitting family. Do it in this order and the saving shows up on the bill.
- Then commit. Only 29% of organizations cover more than half of their eligible spend with commitment discounts (Datadog, State of Cloud Costs 2024). Buy them after right-sizing, so you don't commit to capacity you are about to remove.
- Clean up the long tail. 83% of organizations still run previous-generation EC2 instances, spending about 17% of their EC2 budget on them, and gp2 volumes are still 58% of the average organization's EBS spend (Datadog, State of Cloud Costs 2024). gp3 is up to 20% cheaper per GB (AWS, Amazon EBS General Purpose volumes). See finding orphaned volumes safely.
Who does what
From infrastructure to an approved change
- KubeOn, automated
- Your decision
- Your systems and people
- Only path to a change or purchase
- Step 1, Collect, by Agent + billing:
The read-only agent sends cluster snapshots. Billing exports or rate cards provide the cost basis.
- Step 2, Allocate, by KubeOn hub:
The hub assigns cost to workloads and teams, separates shared and idle spend, and reconciles the totals.
- Step 3, Review, by Your reviewers:
A proposal brings together the saving, evidence, risk and patch. An authorized reviewer records a decision.
- Step 4, Apply + verify, by Your pipeline:
Your pipeline applies the approved patch. Your team checks service health and measures the result.
KubeOn reads and recommends. The only arrow into your clusters starts in your pipeline.
Read the workflowWho does what
From infrastructure to an approved change
- 01
Collect
The read-only agent sends cluster snapshots. Billing exports or rate cards provide the cost basis.
Agent + billing
- 02
Allocate
The hub assigns cost to workloads and teams, separates shared and idle spend, and reconciles the totals.
KubeOn hub
- 03
Review
A proposal brings together the saving, evidence, risk and patch. An authorized reviewer records a decision.
Your reviewers
- 04
Apply + verify
Your pipeline applies the approved patch. Your team checks service health and measures the result.
Your pipeline
KubeOn reads and recommends. Your team and pipeline change the cluster.
Read the workflowHow KubeOn handles it
KubeOn starts from the billed, net amortized cost of every node, so commitments and discounts are already in the numbers. Each pod is charged on max(request, usage) for each hour it ran. Infrastructure idle, workload idle and shared platform namespaces get their own columns, and billed resources that cannot be attributed are listed as unallocated with their resource IDs. Every cluster reconciles to its bill daily.
Savings are computed from the same ledger: container requests from 7 days of p95 usage, node groups from 14 days of peak requests, volumes from their state and billing. Each saving becomes a change proposal with evidence, a risk level and a patch, which your team approves and your pipeline applies. See the savings methodology for every rule and threshold.
Is 83% idle typical for every cluster?
It is an aggregate across Datadog's customers, not a benchmark for any one cluster. Measure your own: compare each cluster's billed node cost with the cost of what pods requested and what they used, over at least a week.
Why not just buy more Savings Plans or reservations?
Commitments lower the price of capacity; they don't reduce how much capacity you run. Buying them before right-sizing locks in the discount on capacity you are about to remove.
Does lowering requests save money immediately?
Not by itself. A lower request moves cost from workload idle into cluster idle until node groups scale down or pods pack more tightly. Compare the cluster's bill before and after.
Where do shared costs like kube-system go?
Into their own column. Who pays for them is a policy decision (the platform team, every team in proportion, or a mix), and it should be dated so a change next quarter does not rewrite this one.
Sources
- Flexera, State of the Cloud 2025 (press release, 19 March 2025)
- FinOps Foundation, State of FinOps 2025
- Datadog, State of Cloud Costs 2024
- CNCF, Cloud Native and Kubernetes FinOps microsurvey (December 2023)
- InfoQ, CNCF survey on Kubernetes overspend (March 2024)
- Kubernetes, Resource management for pods and containers
- AWS, Amazon EBS General Purpose volumes