What a million tokens really costs

AI spend has become a FinOps problem in a single year. In the FinOps Foundation's 2025 survey, 63% of practitioners manage AI spend, up from 31% the year before (FinOps Foundation, State of FinOps 2025). In Flexera's 2025 survey, 72% of organizations use generative AI public cloud services, up from 47% (Flexera, State of the Cloud 2025 (press release, 19 March 2025)). And the price of the underlying capability keeps falling: Andreessen Horowitz estimated that for an LLM of equivalent performance, the cost is decreasing by 10x every year (Andreessen Horowitz, Welcome to LLMflation (November 2024)).

Falling list prices and rising bills are not a contradiction. Usage grows faster than prices fall, and the list price per million tokens is only one input to what a million tokens actually costs you.

Why the list price is not your price

• Output costs more than input. Most models price output tokens at a multiple of input tokens, so a chatty prompt and a verbose answer have very different costs per request.
• Cached tokens are cheaper, or free on provisioned capacity. Repeated prompt prefixes can be served from a cache; on Azure's provisioned deployments, cached tokens don't consume capacity at all.
• Retries and fallbacks are billed too. A request retried after a timeout, or sent to a second provider after a 429, is paid for twice.
• System prompts ride on every call. A long instruction block multiplied by millions of requests is often the largest single item.
• Provisioned capacity is billed by the hour. That is the big one, below.

Provisioned capacity: billed whether or not you use it

On Azure, a provisioned deployment holds a fixed amount of model processing capacity, measured in provisioned throughput units (PTUs), exclusively for you. It is billed at an hourly rate per PTU "regardless of the number of tokens consumed", from the moment the deployment is created until it is deleted. Azure Reservations lower the effective hourly rate in exchange for a 1-month or 1-year commitment, and are bought separately for Global, Data Zone and Regional deployment types. A reservation is a discount, not a guarantee of capacity (Microsoft Learn, Provisioned throughput for Foundry Models).

Amazon Bedrock and Google Vertex AI sell provisioned throughput on the same principle. The consequence is the same everywhere: the cost per million tokens of provisioned capacity is the hourly bill divided by the tokens you actually pushed through it.

effective cost per 1M tokens =  (pay-as-you-go charges + provisioned hourly charges + amortized reservations)  / (tokens served / 1,000,000)

When does provisioned capacity pay off?

Take a deployment with a provisioned commitment costing $18,000 a month, and a pay-as-you-go price for the same model of $2.50 per million tokens. The provisioned deployment's cost per million tokens depends only on volume:

Below the break-even volume, the reservation is buying latency, not savings. Above it, every extra token is nearly free until the capacity is full, which is why the next question is how much capacity to provision.

Size from per-minute demand, not the monthly total

• Collect utilization per minute: capacity is sized for minutes, and a monthly average hides the peaks.
• Size to a percentile of per-minute demand that matches the latency you need, and let the rest spill over to pay-as-you-go. On Azure, spillover can send requests that hit a fully used provisioned deployment to a standard deployment automatically.
• Wait for enough history: KubeOn needs at least 7 days of per-minute data to size a deployment, and 14 days before it counts toward reservation coverage.
• Compare reservations with deployed capacity per pool. Global, Data Zone and Regional are separate quota pools and separate reservation products; a reservation in one cannot cover a deployment in another.

Route every request to the cheapest place that meets its needs

Once capacity is sized, routing decides whether it gets used. Send each request to provisioned capacity first, spill over to pay-as-you-go when it is busy, and fall back to another provider on errors. Give each team its own key, budget and rate limit, so a runaway experiment hits its own ceiling rather than everyone's.