Model Serving Cost Calculator
Turn token assumptions into a recurring budget.
← Back to Compute College
Compute College
What is Cost per Million Tokens?
Cost per million tokens is how hosted AI APIs price inference — usually with input and output tokens priced separately.
Plain-English definition
Cost per million tokens is the price of running a model expressed per million tokens, the way most hosted AI APIs quote inference. It translates GPU-level cost into the unit API buyers actually see, and it usually prices input (prompt) tokens and output (generated) tokens separately.
Memory trick: Tokens are the meter; cost per million tokens is the price per unit — and output usually spins the meter faster.
Why it matters
Most AI products are billed and budgeted in tokens, not GPU-hours. Cost per million tokens connects model usage directly to spend, and because output tokens usually cost more than input tokens, it exposes where the bill actually comes from.
- It is the unit hosted APIs bill in, so it maps usage straight to spend.
- Input and output tokens are usually priced differently, and output often dominates the bill.
- It bridges API buyers, who think in tokens, and infrastructure buyers, who think in GPU-hours.
Simple example
Suppose a feature sends 1,000 input tokens and generates 2,000 output tokens per request, at illustrative rates of $3 per million input tokens and $15 per million output tokens. That is (1,000 × $3 + 2,000 × $15) ÷ 1,000,000 = about $0.033 per request, or roughly an illustrative $33,000 a month at one million requests.
- Multiply token counts by per-million rates, keeping input and output separate.
- Output usually drives the bill, so estimate expected output length carefully.
- Treat per-token rates as illustrative unless taken from a current provider quote.
Example figures are illustrative calculations, not current quoted market prices.
Common mistake
Comparing a single blended per-token rate across providers. Input and output are usually priced differently, context length and features change the effective cost, and a low headline rate can hide expensive output pricing.
Practical takeaway
What you can do with this
Budget from expected input and output token volumes at separate rates, then sanity-check against GPU-hour cost if you self-host.
- Buyers: estimate input and output tokens per request separately and apply each rate.
- Founders and analysts: track cost per completed task, not just per token, since one task can take many tokens.
- Compare against a self-hosted GPU-hour estimate to decide between an API and self-hosting.
- Treat per-token rates as illustrative until taken from a current provider quote.
- Keep observed rates separate from modeled per-request costs.
Decision check: compare providers on input and output rates together with your expected token mix, not on a single headline number.
Compute College
Turn the lesson into a number
Use the GPU-Hour Cost Calculator, AI Training Cost Calculator, or Model Serving Cost Calculator.
Compute College track
Model Costs
Step 3 of 7: What is cost per million tokens