NIST AI Risk Management Framework
Primary guidance for measuring AI performance and managing risk across the system lifecycle.
← Back to Compute College
Compute College
Model selection for the workload
Choose a model by task quality, latency, token economics, reliability, and deployment constraints.
Plain-English definition
Model selection is the process of matching a model’s measured behavior and operating cost to a workload’s requirements. The best model is not necessarily the largest or cheapest per token; it is the option that meets the required outcome within latency, reliability, privacy, capacity, and budget limits.
Memory trick: Choose the model that fits the work, not the model with the loudest score.
Why it matters
A benchmark score or list price is only one input. A model that needs more context, produces longer answers, or retries more often can cost more per accepted task. A smaller model with a clear fallback may be the better production choice when it meets the rubric.
- Evaluate candidate models on the same task and acceptance criteria.
- Include tokens, latency, review, retries, and failure modes in the comparison.
- Keep a fallback or routing policy explicit rather than assuming the primary model is always available.
Simple example
A team compares two models on 200 support cases. Model A passes 91% on the rubric at 1.2 seconds p95; Model B passes 94% at 3.8 seconds p95 and doubles output tokens. The choice depends on the product’s latency and cost limits, not the score alone.
- The comparison uses the same cases and rubric.
- The tail-latency and token difference may matter more than three accuracy points.
- A human-review path can change the acceptable threshold.
Example figures are illustrative calculations, not current quoted market prices.
Current example
Primary reference
This source explains the method or risk boundary. It does not replace testing the actual model, data, provider, and workload conditions in production.
Source discipline: examples on this page are illustrative; measure the real workload before making a release or capacity decision.
Common mistake
A leaderboard or token price cannot select the model for you. Production fit depends on the task, context, concurrency, safety boundary, and accepted-result economics.
Practical takeaway
What you can do with this
Create a candidate matrix with the task rubric, token range, latency, review rate, failure categories, privacy constraints, and cost per accepted result. Test at least one smaller or simpler option.
- Keep model settings and context policy comparable.
- Test the failure cases that matter most to users.
- Document why the chosen model is sufficient, not only why alternatives were rejected.
Decision check: can you explain which requirement the selected model satisfies and what tradeoff the team accepted?
Compute College learning path
AI Engineering
Step 40 of 48: Model selection for the workload