Google SRE workbook
Primary reliability guidance on service objectives, error budgets, overload, and safe operational response.
← Back to Compute College
Compute College
Reliability and operational design
Design fallbacks, timeouts, retries, and service boundaries that keep AI workloads useful during failure.
Plain-English definition
AI operational design defines what the system does when a model, provider, tool, network, or data source is slow or unavailable. Reliability comes from bounded retries, timeouts, fallbacks, queues, cached or partial results, and clear user-visible degradation—not from assuming every model call succeeds.
Memory trick: Reliability is a designed failure path.
Why it matters
AI services depend on multiple variable components. Retrying everything can amplify an outage and exhaust capacity; failing silently can damage trust. Explicit degradation protects both users and the market resources the workload consumes.
- The design makes a hidden tradeoff measurable before it reaches users.
- A clear boundary reduces retries, unsafe actions, or unnecessary capacity.
- The operating choice should be tested against the workload rather than assumed from a demo.
Simple example
If retrieval times out, an assistant does not invent an answer. It tells the user that source evidence is unavailable, offers a retry, and records the event. If the primary model is unavailable, a smaller approved fallback handles only tasks within its rubric.
- The example names the input, decision, and expected boundary.
- A deterministic or human checkpoint owns the part a model should not guess.
- The result can be measured with quality, latency, safety, and cost metrics.
Example figures are illustrative calculations, not current quoted market prices.
Current example
Primary reference
This source explains the method or risk boundary. It does not replace testing the actual model, data, provider, and workload conditions in production.
Source discipline: examples on this page are illustrative; measure the real workload before making a release or capacity decision.
Common mistake
Unlimited automatic retries are not reliability. They can increase latency, duplicate actions, and turn a partial outage into a capacity event.
Practical takeaway
What you can do with this
Document each dependency’s timeout, retry budget, fallback, user message, and audit behavior. Test provider errors and partial tool failures before launch.
- Start with a narrow workflow and explicit failure boundary.
- Instrument the path before optimizing it.
- Review the design whenever model, tool, traffic, or policy changes.
Decision check: when a dependency fails, does the system preserve safety and explain the degraded result without amplifying load?
Compute College learning path
AI Engineering
Step 46 of 48: Reliability and operational design