OpenAI prompt caching documentation
Official documentation on cached prompt prefixes and the conditions that affect cached input processing.
← Back to Compute College
Compute College
Token, context, and cache economics
Connect tokens, context size, caching, and reuse to AI workload cost, latency, and capacity.
Plain-English definition
Token economics describes how input tokens, output tokens, repeated context, cached prefixes, and retrieval choices contribute to an AI workload. Caching can reduce repeated processing or price in some systems, but it does not make irrelevant context free: cache storage, invalidation, latency, and provider terms still matter.
Memory trick: Every repeated token is a workload decision.
Why it matters
Prompt and context decisions repeat on every request. A small increase in input or output tokens can become a large monthly workload when traffic is high. Measuring reuse and accepted results helps teams optimize the right cost instead of shortening text blindly.
- Input and output tokens can have different prices and latency effects.
- Repeated instructions or context may be candidates for caching.
- The useful denominator is often accepted task, not request or token alone.
Simple example
An assistant receives 100,000 requests per month with 3,000 input tokens and 500 output tokens. If a shared 1,500-token instruction and policy prefix can be reused safely, the team can compare cache savings with invalidation complexity and the cost of sending changing context.
- Monthly volume makes per-request differences material.
- Only stable, authorized context should be reused.
- The output and retry path still contribute to cost.
Example figures are illustrative calculations, not current quoted market prices.
Current example
Primary reference
This source explains the method or risk boundary. It does not replace testing the actual model, data, provider, and workload conditions in production.
Source discipline: examples on this page are illustrative; measure the real workload before making a release or capacity decision.
Common mistake
The largest context window is not automatically the best economic choice. Extra context can add tokens, latency, memory pressure, and irrelevant evidence without improving the accepted result.
Practical takeaway
What you can do with this
Create a token budget for one workload. Measure stable versus changing context, output length, retries, cache hits, latency, and accepted-task cost before optimizing.
- Set a maximum useful output length.
- Cache only stable and authorized prefixes.
- Test whether shorter or better-ranked context preserves the rubric score.
Decision check: can you show which tokens improve the accepted outcome and which are repeated cost without measurable value?
Compute College learning path
AI Engineering
Step 44 of 48: Token, context, and cache economics