Anthropic: Building effective agents
Primary guidance on simple workflows, tool use, orchestration, and adding autonomy only when it creates measured value.
← Back to Compute College
Compute College
Routing, fallbacks, and model portfolios
Use task-aware model routing and approved fallbacks to balance quality, latency, resilience, and cost.
Plain-English definition
Model routing sends different requests to different approved models or deployments based on task, risk, context, latency, or capacity. A model portfolio is the set of primary, fallback, and specialized options plus the policy that chooses among them.
Memory trick: A portfolio is a policy, not a pile of models.
Why it matters
One model rarely dominates every task and operating condition. Routing can use a smaller model for simple work and reserve a stronger model for difficult or high-risk cases, but the policy must be evaluated for quality, fairness, availability, and cost.
- The design makes a hidden tradeoff measurable before it reaches users.
- A clear boundary reduces retries, unsafe actions, or unnecessary capacity.
- The operating choice should be tested against the workload rather than assumed from a demo.
Simple example
A support system routes short classification requests to a smaller model, sends ambiguous or policy-sensitive cases to a stronger model, and falls back to a queued human review when both are unavailable. The routing decision is logged and tested like any other product behavior.
- The example names the input, decision, and expected boundary.
- A deterministic or human checkpoint owns the part a model should not guess.
- The result can be measured with quality, latency, safety, and cost metrics.
Example figures are illustrative calculations, not current quoted market prices.
Current example
Primary reference
This source explains the method or risk boundary. It does not replace testing the actual model, data, provider, and workload conditions in production.
Source discipline: examples on this page are illustrative; measure the real workload before making a release or capacity decision.
Common mistake
Token price alone is a poor routing policy. A cheaper model that produces more retries or unsafe escalations may have a higher cost per successful outcome.
Practical takeaway
What you can do with this
Define routing rules, confidence or task signals, fallback order, capacity limits, and audit metrics. Compare the portfolio with a single-model baseline.
- Start with a narrow workflow and explicit failure boundary.
- Instrument the path before optimizing it.
- Review the design whenever model, tool, traffic, or policy changes.
Decision check: can the portfolio explain why each request went to a model and what happens when that option is unavailable?
Compute College learning path
AI Engineering
Step 45 of 48: Routing, fallbacks, and model portfolios