OpenTelemetry documentation
Primary observability guidance for tracing distributed operations and connecting latency and failure evidence across components.
← Back to Compute College
Compute College
Agent observability and evaluation
Instrument agent trajectories so teams can explain decisions, failures, latency, and cost.
Plain-English definition
Agent observability means keeping enough of a run to explain what the system saw, decided, called, returned, and why it stopped. Evaluation uses that record to check the task result, safety, reliability, and cost without putting sensitive data into every log.
Memory trick: Observe the trajectory; protect the content.
Why it matters
A final answer rarely explains why an agent failed. Trace-level evidence reveals repeated searches, bad arguments, stale context, permission denials, and expensive paths. Without it, teams cannot improve or price the system reliably.
Simple example
For every task, the system records a redacted trace ID, model version, prompt version, tool names and durations, token counts, approval events, stop reason, final outcome, and cost estimate. Sensitive document contents are not copied into general logs.
Example figures are illustrative calculations, not current quoted market prices.
Current example
Source material
Common mistake
Logging every prompt and tool result creates risk without a data policy. Useful traces need correlation and metrics, but sensitive content must be minimized, protected, or redacted.
Practical takeaway
What you can do with this
Define a trace schema with IDs, versions, timings, calls, budgets, outcomes, and redaction rules. Build dashboards for cost per accepted task, trajectory length, p95 latency, and safety escalations.
Decision check: can an operator explain a failed or expensive task without exposing more user data than necessary?
Compute College learning path
AI Engineering
Step 36 of 48: Agent observability and evaluation