OpenAI evaluation best practices
Official evaluation guidance on defining criteria, comparing outputs, and improving the evaluation loop.
← Back to Compute College
Compute College
Define the task, ground truth, and scoring rubric
Turn an AI evaluation goal into a task definition, reference answer, and scoring rubric that reviewers can apply consistently.
Plain-English definition
Ground truth is the reference information or decision an evaluation uses to judge an output. A scoring rubric describes how to award credit, reject a result, or mark a case uncertain. Together, the task definition, ground truth, and rubric turn subjective review into a documented measurement process.
Memory trick: Ground truth is the reference; the rubric is the measuring stick.
Why it matters
A vague rubric creates noisy scores: two reviewers can see the same answer and apply different standards. Clear criteria reveal whether a model is missing evidence, violating a constraint, or simply facing an ambiguous case. That clarity keeps teams from optimizing a number that does not represent user value.
- Reference answers should record acceptable variation, not force one phrasing.
- Rubric levels make partial credit and serious failures explicit.
- Uncertain or disputed cases should be labeled rather than quietly folded into a score.
Simple example
For a policy-answering task, a four-point rubric gives one point for answering the question, one for citing the correct policy passage, one for handling an exception, and one for avoiding unsupported advice. A response can be fluent and still score zero on evidence.
- Each point tests a distinct requirement.
- The rubric distinguishes a harmless omission from an unsafe claim.
- Reviewers can record “not applicable” when a case does not contain an exception.
Example figures are illustrative calculations, not current quoted market prices.
Current example
Primary reference
This source explains the method or risk boundary. It does not replace testing the actual model, data, provider, and workload conditions in production.
Source discipline: examples on this page are illustrative; measure the real workload before making a release or capacity decision.
Common mistake
A single exact reference sentence is too narrow for many tasks. The rubric should judge required facts, evidence, and constraints rather than surface wording when several answers could be acceptable.
Practical takeaway
What you can do with this
Choose one workflow and write a rubric with three to five independently scorable requirements. Have two reviewers score the same cases, compare disagreements, and revise the rubric before using it as a release gate.
- Define what counts as a critical failure.
- Allow uncertainty where the source data is ambiguous.
- Keep examples of passing and failing outputs with the rubric.
Decision check: do independent reviewers reach the same decision for the same output using only the written rubric?
Compute College learning path
AI Engineering
Step 20 of 48: Define the task, ground truth, and scoring rubric