OpenAI Evals guide
Official guidance on organizing repeatable evaluations for model and application behavior.
← Back to Compute College
Compute College
Why AI applications need evaluations
Learn why representative evaluations are necessary before shipping or changing an AI application.
Plain-English definition
An AI evaluation is a repeatable way to compare model or application behavior against a defined task and acceptance standard. It can combine reference answers, rules, human review, model-based grading, and workload measurements. An evaluation is not a single impressive example; it is evidence across cases that matter.
Memory trick: A demo shows possibility; an evaluation shows repeatability.
Why it matters
AI outputs vary with wording, context, model updates, and data. Without an evaluation, a change can look better in a demo while lowering accuracy on important cases, increasing unsafe behavior, or raising retries and cost. Evaluation gives a team a release decision that can be inspected later.
- A test set exposes edge cases that a hand-picked demo hides.
- Quality and workload metrics show whether an improvement is worth its latency and token cost.
- A baseline makes model, prompt, retrieval, and infrastructure changes comparable.
Simple example
A support classifier is evaluated on 100 labeled tickets: 60 ordinary cases, 20 ambiguous cases, 10 policy-sensitive cases, and 10 out-of-scope cases. The release record includes category accuracy, unsupported-action rate, p95 latency, tokens, and human-review rate.
- The mix reflects the product rather than only the easiest traffic.
- Safety and escalation behavior are measured separately from label accuracy.
- The same baseline can be rerun after a prompt or model change.
Example figures are illustrative calculations, not current quoted market prices.
Current example
Primary reference
This source explains the method or risk boundary. It does not replace testing the actual model, data, provider, and workload conditions in production.
Source discipline: examples on this page are illustrative; measure the real workload before making a release or capacity decision.
Common mistake
One good answer proves only that the application can work once. A demo demonstrates possibility; a representative evaluation tests repeatability, failure boundaries, and cost.
Practical takeaway
What you can do with this
Write a first evaluation contract for one workflow. Define the task, cases, pass criteria, unacceptable failures, reviewer process, and workload metrics before changing the prompt.
- Start with a small representative set, then add every important production failure.
- Separate quality, safety, and operational metrics.
- Store the model, prompt, context policy, and date beside each result.
Decision check: could another person reproduce the comparison and explain why the new version passed or failed?
Compute College learning path
AI Engineering
Step 19 of 48: Why AI applications need evaluations