How to monitor AI agents in production: trace the outcome

Start at the business outcome

“Done” in the chat window is not enough. For a support agent, check that the draft exists, belongs to the right ticket, and reaches the person who can approve it. Monitoring needs to follow the work into the destination system, not stop at a successful model response.

OpenTelemetry traces describe work through spans with timing, attributes and relationships. This is a useful foundation for following one logical task through a model request and several tool calls. Trace concepts.

Define the metrics before drawing a dashboard

Use explicit denominators. If 100 tasks entered the workflow and 90 reached a verified completion, the completion rate is 90%. If 20 of those completions needed review, report that intervention count separately; do not silently redefine success as “no human involved.” These numbers are a worked example, not site or client measurements.

Failed tasks need their own duration and age view. A latency percentile calculated only from completed tasks can conceal a growing queue of stuck work. Record cancellation, expiry and unresolved states explicitly.

Alert on an action someone can take

For every alert, name the owner, the likely symptom and the next action. A rise in authentication failures should route to the person who maintains the integration. A rise in policy denials may indicate a changed workflow, a bad prompt or abusive input; the right response is investigation, not automatically relaxing the gate.

Choose thresholds from a baseline and the impact of failure. Avoid borrowing a universal “healthy agent” score. A workflow that drafts an internal note and one that changes account access require different operating policies.

Keep traces useful without leaking payloads

Prefer stable task IDs, tool names, timing and error categories. Redact tokens and personal content before exporting telemetry. Limit access and retention. If you need a sample payload to debug, obtain it through the approved data-handling process rather than enabling unrestricted capture.

A recovery runbook should explain how to pause new work, identify a partial write, reconcile the destination state and resume safely. Validate that runbook during a controlled failure test; a document alone does not establish recoverability.

Put monitoring beside offline evaluation

A fixed evaluation suite catches regressions against known tasks. Production monitoring catches changing inputs and dependencies. Keep both, and add representative failure cases back into the evaluation dataset after incidents. See the evaluation guide and security boundary guide.

Build the first incident view

Start with a list of unresolved logical tasks, sorted by age and owner. Include the last completed step, the current error category and a link to the redacted trace. This gives the operator a concrete queue to work through rather than a dashboard of averages that cannot identify the affected customer request.

For a repeated failure, compare the time of the first occurrence with credential changes, deployments and upstream service incidents. Preserve the original error category while adding your interpretation. Replacing the source error with a model-written explanation can remove the clue needed to diagnose the dependency.

Once the incident is resolved, add a representative case to the offline evaluation suite and verify the recovery path. Record whether the fix addressed the underlying cause or only reduced its symptoms. Monitoring becomes more useful when each incident improves a test, a runbook or an ownership boundary.

Sources and review notes

OpenTelemetry trace concepts checked October 14, 2026. Metric definitions, the 100-task arithmetic example and the runbook guidance are original editorial methods. No production telemetry or claimed performance improvement is included.