OpenAI Structured Outputs guide
Official documentation for schema-constrained responses and the need for application-side validation.
← Back to Compute College
Compute College
Deterministic graders and rule-based checks
Use code and explicit rules for the parts of AI evaluation that do not require model judgment.
Plain-English definition
A deterministic grader evaluates an output with a repeatable rule whose result does not depend on another model call. Examples include schema validation, exact or normalized matching, required-field checks, citation presence, range checks, and permission rules. Deterministic checks should own requirements that software can verify directly.
Memory trick: Rules belong in code when code can know the answer.
Why it matters
A model-based grader can be useful for nuanced language, but it adds cost, latency, and its own variability. Deterministic checks are easier to audit and make critical failures explicit. Combining the two lets a system use judgment where needed without asking a model to enforce a rule that code can enforce better.
- Schemas catch malformed structure before downstream code acts.
- Rules make security, ranges, enums, and required evidence auditable.
- Deterministic checks are cheap to rerun during regression tests.
Simple example
A claims extractor first checks that the response is valid JSON, contains only allowed fields, quotes evidence from the source, and uses a confidence value between 0 and 1. Only then does a model-based grader assess whether the explanation actually supports the claim.
- The cheap checks run before the expensive semantic grader.
- A failed structural check produces a specific error rather than a vague low score.
- The application can route unsupported claims to review.
Example figures are illustrative calculations, not current quoted market prices.
Current example
Primary reference
This source explains the method or risk boundary. It does not replace testing the actual model, data, provider, and workload conditions in production.
Source discipline: examples on this page are illustrative; measure the real workload before making a release or capacity decision.
Common mistake
Code should enforce rules that have exact answers. If the requirement is an allowed value, a required field, a range, or a permission, use a deterministic check before asking a model.
Practical takeaway
What you can do with this
List the evaluation criteria for one workflow and mark each as deterministic, semantic, or human. Implement the deterministic checks before tuning the semantic grader.
- Fail closed for missing permissions or required evidence.
- Return actionable validation errors for logs and review.
- Test validators with malformed and adversarial outputs.
Decision check: if the model returned a confident but invalid value, would code stop it before a user or tool sees it?
Compute College learning path
AI Engineering
Step 22 of 48: Deterministic graders and rule-based checks