Updated
Instrument AI agent workflows with useful traces, outcome metrics, privacy controls and failure alerts that explain what happened beyond model responses.
AI agent monitoring should connect a user's request to the model calls, tools, policy decisions and verified business outcome that followed. A dashboard showing token usage and successful API responses cannot explain whether an agent changed the wrong record, repeated an action or quietly abandoned a task.
Use an invoice-processing assistant as a running example. It receives a supplier document, extracts line items, checks a purchase order and creates a draft payable. Monitoring must distinguish a draft successfully created from an invoice merely described as processed. The finance database is the source of truth for the action; the assistant's final sentence is not.
Start with the question an incident responder must answer
Design telemetry around a concrete investigation: “Why was invoice 782 created twice after yesterday's release?” The responder needs to connect the document, two workflow attempts, their tool calls and the resulting database records.
Give a logical workflow a durable identifier. Give each execution attempt its own identifier. Link tool requests to the attempt and store a separate business-action identifier for operations that may be retried. These identifiers serve different purposes: a restarted attempt can belong to the same workflow and should not necessarily create another payable.
Capture release version, prompt version, tool contract version and model identifier. Without them, comparing yesterday's failure with today's successful replay can conceal the change that caused the incident.
Build traces around meaningful boundaries
OpenTelemetry's tracing guide describes spans and trace relationships for following work across components. In the invoice workflow, useful boundaries include document retrieval, extraction, purchase-order lookup, policy evaluation and draft creation.
Do not make every sentence or internal reasoning fragment a span. Instrument operations that consume resources, cross services or make decisions relevant to the outcome. Record the observed result and duration, not an imagined explanation of the model's thinking.
Keep asynchronous boundaries visible. A queue job may begin after the original web request ends; preserve the appropriate correlation or span link so the responder can follow it. For a large batch, one failing invoice should be identifiable without treating the entire batch as a single opaque operation.
The OpenTelemetry GenAI conventions repository is the current home for GenAI-specific conventions. Pin the conventions supported by your instrumentation and review their stability before adopting attribute names. Do not build a permanent dashboard on a field merely because an older tutorial presents it as settled.
Record facts that explain action failures
For a consequential tool call, collect a small structured record:
Operation name and tool contract version.
Workflow, attempt and business-action identifiers.
Authenticated tenant reference, using an appropriate pseudonymous value.
Whether validation and authorization passed.
Number of retries and their classified causes.
Start time, end time and timeout budget.
The destination service's request identifier.
The confirmed result record or reconciliation status.
For the duplicate invoice, these facts can reveal that the first create succeeded but its response was lost. The next attempt then created another record with a new action identifier. That is an execution defect even if both model calls produced valid JSON.
Avoid logging full access tokens, document bodies or payment details just to make troubleshooting convenient. Retain sensitive evidence separately with explicit access and deletion rules when it is genuinely necessary. Telemetry must not become an uncontrolled copy of the customer database.
Define outcome metrics separately from infrastructure metrics
Measure extraction validity, purchase-order match quality, duplicate drafts and human correction separately. A workflow that escalates an ambiguous invoice can be working correctly; a workflow that creates an incorrect draft quickly can be failing.
Create an outcome taxonomy that operations staff can apply consistently. For example: completed and verified, pending external input, handed to a person, rejected by policy, failed before a write, and outcome unknown after a write. “Unknown” deserves its own queue because assuming failure may cause harmful retries.
Report denominators with every rate. A duplicate rate calculated over all chat messages obscures the risk among actual create operations. Group results by relevant workflow version and document type, and keep sample sizes visible before drawing conclusions from a small slice.
For quality checks beyond telemetry, use a representative evaluation set. Our RAG evaluation framework shows how to isolate retrieval and answer defects when the agent depends on document evidence.
Make cost and latency interpretable
Track total workflow time alongside model and tool time. A slow workflow may spend most of its time waiting for a purchase-order API or a human decision, even when model generation is fast.
Measure input, output and cached token usage according to the provider's actual reporting semantics. Attach a dated pricing configuration to cost estimates; avoid presenting an inferred charge as the final invoice. Include retried calls, document extraction and external tool charges when those contribute to the service's cost.
Compare cost per verified completed invoice, with exceptions reported separately. A lower average model bill can be misleading if the new configuration creates more cases requiring manual repair. The model routing guide discusses evaluating cheaper routes against the quality requirement they must still meet.
Alert on actionable symptoms
Start with a few alerts tied to a response owner. Examples are a burst of duplicate action identifiers, an increasing unresolved-write queue, repeated authorization failures, or a sustained change in completion time against the team's service objective.
Do not wake an engineer for every expected policy rejection. Route document-quality problems to the people who can fix the inputs. Include a trace link, affected workflow version and recovery procedure in each actionable alert.
Sampling also needs deliberate treatment. Keep lightweight counters across all requests while sampling expensive traces. Preserve enough evidence for failed or consequential actions according to your retention policy, and account for the fact that a sampled trace set cannot directly provide every population-level rate.
Run an observability acceptance drill
Before release, inject a lost tool response and a malformed purchase-order result. Ask a teammate who did not build the workflow to diagnose both cases using only the operational dashboard and approved evidence stores.
They should identify the affected invoice, whether a write occurred, which version ran and what recovery is safe. If they need access to a developer's terminal history, the instrumentation is incomplete. Connect the drill to the team's AI incident response plan.
Should prompts and responses always be logged?
No. Log metadata by default and collect content only where its diagnostic value justifies the access, retention and privacy implications. Redact before export, and test that the redaction works on actual payload shapes.
Does tracing prove an agent is accurate?
It explains observed execution. Accuracy still requires source verification, evaluations and, where appropriate, human review. A complete trace can faithfully document a wrong decision.
Written by
Moueen Togarvi
Founder & CEO at Voquarn Code, focused on product engineering, search growth, and practical AI systems.
