Engineering
Why cost recommendations don't get shipped
Most cost tools find savings. Few savings reach production. What engineers need before they change a running service, and what changes when a saving arrives as a reviewed patch instead of a dashboard row.
KubeOn Engineering
October 9, 20264 min read
Finding savings is the easy part. Every cost tool lists oversized requests, idle nodes and forgotten volumes. Getting them into production is where cost programs stall. In the FinOps Foundation's surveys, empowering engineers to take action was the top key priority from the survey's start in 2020 until 2024, when reducing waste displaced it for the first time (FinOps Foundation, Key priorities shift in 2024 (February 2024)). In 2025, investment and tooling (time, upskilling, tooling and automation) rose 20% as the path practitioners need to reach their priorities (FinOps Foundation, State of FinOps 2025).
The CNCF's survey on Kubernetes spending points at the same gap from the engineering side: 45% of respondents named a lack of awareness and responsibility at the individual and team level as a cause of overspending, and only 20% had made FinOps operational, actively monitoring and reporting spend (InfoQ, CNCF survey on Kubernetes overspend (March 2024)).
What a recommendation has, and what an engineer needs
| A typical recommendation | What the engineer needs before acting |
|---|---|
| "Reduce CPU request, save $310/month" | The usage data behind it: which window, which statistic, how many samples |
| A single saving figure | The risk: what could break, and how likely that is for this kind of change |
| A link to a console | The exact change, written in the format their repository uses |
| Nothing about rollout | Safe rollout steps, and how to roll back |
| No owner | Which team owns the workload, and who is allowed to approve |
| No follow-up | Whether the change was applied, and whether the bill moved |
Every gap on the right is work the engineer must do before touching a running service, on top of their real work. Most rows never get that time. That is not resistance; it is a reasonable reading of risk against reward.
Turn the saving into a change
The fix is to do that work before the engineer sees the saving. A change proposal carries everything in the right-hand column: scope, current and proposed state, the evidence, a risk level, the monthly saving at billed prices, the patch itself, the risks, and safe rollout steps.
# Evidence: 7 days, 168 hourly samples per pod# CPU p95 410m (request 1500m) -> 492m with 20% headroom# Memory p95 0.7Gi (request 3Gi) -> 0.84Gi with 20% headroom# Saving: about $142/month at your billed node rate# Rollout: one replica first, watch restarts and latency for 24 hoursresources: requests: cpu: 492m memory: 860MiThe engineer's job becomes a review: check the evidence against what they know about the service, approve or reject with a note, and merge the patch through the repository and pipeline they already use. The cluster changes the same way it always does, with the same tests, approvals and change windows.
Keep the queue worth reading
- Set a threshold. A proposal for every $3 saving trains people to ignore the list. In KubeOn, every change worth $25 or more a month gets its own proposal, up to 250 at a time.
- Bundle the long tail. Smaller items are grouped per cluster, or per account and region for volumes, with a per-item evidence list and a $5 a month floor.
- Keep decisions. A rejected proposal stays rejected when proposals regenerate, so nobody reviews the same change twice.
- Retire stale ones. When the data no longer supports a proposal, it is marked superseded rather than left to rot.
- Record everything. Every approval, rejection and applied change is in the audit log with the reviewer and their note.
Measure what shipped, not what was found
"Savings identified" is a vanity number. Track three things instead: how much of the identified saving has a proposal (coverage), how much was approved and applied, and what the bill did afterwards. A lower request moves cost into idle until nodes scale down, so the last number, the cluster's bill before and after, is the one finance will believe.
Who does what
From infrastructure to an approved change
- KubeOn, automated
- Your decision
- Your systems and people
- Only path to a change or purchase
- Step 1, Collect, by Agent + billing:
The read-only agent sends cluster snapshots. Billing exports or rate cards provide the cost basis.
- Step 2, Allocate, by KubeOn hub:
The hub assigns cost to workloads and teams, separates shared and idle spend, and reconciles the totals.
- Step 3, Review, by Your reviewers:
A proposal brings together the saving, evidence, risk and patch. An authorized reviewer records a decision.
- Step 4, Apply + verify, by Your pipeline:
Your pipeline applies the approved patch. Your team checks service health and measures the result.
KubeOn reads and recommends. The only arrow into your clusters starts in your pipeline.
Read the workflowWho does what
From infrastructure to an approved change
- 01
Collect
The read-only agent sends cluster snapshots. Billing exports or rate cards provide the cost basis.
Agent + billing
- 02
Allocate
The hub assigns cost to workloads and teams, separates shared and idle spend, and reconciles the totals.
KubeOn hub
- 03
Review
A proposal brings together the saving, evidence, risk and patch. An authorized reviewer records a decision.
Your reviewers
- 04
Apply + verify
Your pipeline applies the approved patch. Your team checks service health and measures the result.
Your pipeline
KubeOn reads and recommends. Your team and pipeline change the cluster.
Read the workflowWhy not let a tool apply the change automatically?
Some teams do, for some changes. Many enterprises can't or won't give a cost tool write access to production. A reviewed patch gets most of the speed with none of that access, and leaves the decision with the team that is paged when something breaks.
Who should approve a proposal?
The team that owns the workload, or a platform engineer for shared infrastructure. Approval rights should follow ownership, and every decision should be logged.
What if engineers reject most proposals?
Read the rejection notes. Repeated rejections usually point to something the rule can't see, such as a monthly batch peak or a latency target, and that is worth fixing in the rule or in the workload's labels.
How does KubeOn deliver proposals?
In the app and through its API, with the patch as text: Kubernetes YAML, Karpenter NodePool snippets, or snapshot-then-delete scripts for volumes. You copy it into a pull request. KubeOn has no permission to apply anything.