OpenTelemetry documentation
Primary observability guidance for tracing distributed operations and connecting latency and failure evidence across components.
← Back to Compute College
Compute College
Agent observability and evaluation
Instrument agent trajectories so teams can explain decisions, failures, latency, and cost.
Plain-English definition
Agent observability means keeping enough of a run to explain what the system saw, decided, called, returned, and why it stopped. Evaluation uses that record to check the task result, safety, reliability, and cost without putting sensitive data into every log.
Memory trick: Observe the trajectory; protect the content.
Why it matters
A final answer rarely explains why an agent failed. Trace-level evidence reveals repeated searches, bad arguments, stale context, permission denials, and expensive paths. Without it, teams cannot improve or price the system reliably.
- The design makes a hidden tradeoff measurable before it reaches users.
- A clear boundary reduces retries, unsafe actions, or unnecessary capacity.
- The operating choice should be tested against the workload rather than assumed from a demo.
Simple example
For every task, the system records a redacted trace ID, model version, prompt version, tool names and durations, token counts, approval events, stop reason, final outcome, and cost estimate. Sensitive document contents are not copied into general logs.
- The example names the input, decision, and expected boundary.
- A deterministic or human checkpoint owns the part a model should not guess.
- The result can be measured with quality, latency, safety, and cost metrics.
Example figures are illustrative calculations, not current quoted market prices.
Current example
Primary reference
This source explains the method or risk boundary. It does not replace testing the actual model, data, provider, and workload conditions in production.
Source discipline: examples on this page are illustrative; measure the real workload before making a release or capacity decision.
Common mistake
Logging every prompt and tool result creates risk without a data policy. Useful traces need correlation and metrics, but sensitive content must be minimized, protected, or redacted.
Practical takeaway
What you can do with this
Define a trace schema with IDs, versions, timings, calls, budgets, outcomes, and redaction rules. Build dashboards for cost per accepted task, trajectory length, p95 latency, and safety escalations.
- Start with a narrow workflow and explicit failure boundary.
- Instrument the path before optimizing it.
- Review the design whenever model, tool, traffic, or policy changes.
Decision check: can an operator explain a failed or expensive task without exposing more user data than necessary?
Compute College learning path
AI Engineering
Step 36 of 48: Agent observability and evaluation