NIST AI Risk Management Framework
A primary framework for identifying, measuring, and managing risks across the AI lifecycle.
← Back to Compute College
Compute College
Build a practical evaluation dataset
Create a small, representative evaluation set that covers normal traffic, edge cases, and important failures.
Plain-English definition
An evaluation dataset is a maintained collection of inputs, expected requirements, labels, or reference evidence used to test an AI workflow. A useful set samples the real task distribution while deliberately including boundary cases, safety cases, and known failures.
Memory trick: A good evaluation set is a map of the work and the ways it can fail.
Why it matters
A dataset made only from convenient examples overstates quality and gives a team no warning when users behave differently. A maintained evaluation set also becomes institutional memory: when a production failure is added, future prompt and model changes must prove they did not repeat it.
- Normal cases estimate expected performance.
- Boundary and adversarial cases test the system where harm or rework is concentrated.
- Versioning the dataset prevents silent changes from making scores incomparable.
Simple example
A document extractor starts with 80 ordinary documents, 10 documents with missing fields, 5 scanned or malformed inputs, and 5 documents containing prompt-injection text. Each case records the fields that may be extracted and the evidence required for a pass.
- The set includes both frequency and consequence.
- Malformed inputs test graceful failure instead of only successful extraction.
- Injection cases test whether untrusted text changes the instruction hierarchy.
Example figures are illustrative calculations, not current quoted market prices.
Current example
Primary reference
This source explains the method or risk boundary. It does not replace testing the actual model, data, provider, and workload conditions in production.
Source discipline: examples on this page are illustrative; measure the real workload before making a release or capacity decision.
Common mistake
Do not delete difficult cases just because they lower the score. They are often the most valuable part of the dataset; label them clearly and improve the system or the operating boundary.
Practical takeaway
What you can do with this
Create a versioned dataset with a normal-case sample, a boundary-case sample, and a failure registry. Add an owner, source, sensitivity classification, and review date to every case.
- Use synthetic cases only when real data is unavailable and document the limitation.
- Keep sensitive inputs minimized and access-controlled.
- Add a regression case whenever a user-visible failure is found.
Decision check: can you explain which real workload each evaluation case represents and when the set was last refreshed?
Compute College learning path
AI Engineering
Step 21 of 48: Build a practical evaluation dataset