AI Agent Observability: Trace Answers, Actions and Failures

Last updated: 2 October 2026.

A CRM agent says it updated twelve accounts. The API dashboard shows successful requests. Sales finds only nine correct updates, two records in the wrong workspace and one missing change.

A latency chart cannot explain that incident. You need the proposed targets, resolved record identities, workspace context, write results and verification step. The model's confident completion message is only one part of the evidence.

For teams deploying a custom marketing AI agent, observability should be designed around the decisions the operator will need to make: stop the workflow, repair the data, change a tool, adjust instructions or revise the task itself.

What should a trace explain?

Start with one unit of work. For an account-research agent, it might be preparing one account brief. For a lifecycle agent, it might be evaluating one customer for one proposed journey action. Give that unit a stable business identifier independent of any one model request.

A trace describes the operations involved in processing that work. Individual spans represent operations with a beginning and end. Events can record a meaningful point within an operation, such as approval received. OpenTelemetry also supports links between spans, including work that continues asynchronously. OpenTelemetry traces.

For an account brief, instrument the meaningful stages:

1. Accept the request and establish the workspace and authorized scope.
2. Resolve the account identity.
3. Retrieve CRM and approved external evidence.
4. Generate the proposed brief.
5. Validate required claims and fields.
6. Save the draft in the intended destination.
7. Verify that the expected artifact exists.

Do not create a span for every trivial line of code. The useful boundary is a step that can fail differently, incur meaningful cost or change the explanation of the outcome.

A trace cannot reveal private internal model reasoning that the provider does not expose. It can show the actual inputs, outputs, tool calls and application decisions you captured. Label generated explanations as explanations, not proof of the model's hidden thought process.

Use identifiers that survive retries and handoffs

Keep three identities separate: the business task, a particular execution attempt and each consequential action. One task may have several attempts. One attempt may perform multiple actions. A timeout can cause several technical requests for the same intended action.

If each retry receives a new business-task identity, reports inflate volume and obscure repeated failures. If every action shares one undifferentiated ID, you cannot identify which write needs reconciliation.

Record a workspace or tenant identifier at every relevant boundary. Use opaque internal identifiers in telemetry when possible, with access-controlled lookup for investigation. Names and email addresses are unnecessary as metric labels and can expose customer information.

LangSmith supports metadata and tags for organizing traces. Its thread documentation specifically warns that thread metadata must reach child runs for complete filtering and aggregation of thread usage. Verify propagation in your own asynchronous paths rather than assuming a top-level tag appears everywhere. LangSmith metadata, thread configuration.

For a queue handoff, include a safe correlation reference in the job envelope and link the worker's execution back to the initiating task. Do not use that reference as authorization. Knowing a task ID must not grant access to another workspace's records.

Define a small business event contract

Model and tool instrumentation will supply technical details, but it does not know what your team considers successful. Add an explicit business outcome record.

This original example is an application event, not an OpenTelemetry or vendor schema:

{
  "event_type": "account_brief_completed",
  "business_task_id": "brief-task-1042",
  "attempt_id": "attempt-2",
  "workspace_id": "workspace-example",
  "agent_release": "account-brief-v7",
  "prompt_version": "brief-prompt-v4",
  "policy_version": "evidence-policy-v2",
  "source_snapshot_id": "crm-snapshot-31",
  "target_record_id": "account-508",
  "destination_artifact_id": "brief-908",
  "technical_status": "completed",
  "business_status": "needs_review",
  "reason_code": "missing_contract_date",
  "verified_destination": true,
  "human_review_seconds": 75
}

The example deliberately separates a completed execution from an acceptable brief. The application saved the artifact successfully, but a missing source field means the result still needs review.

Define allowed outcome states and reason codes with the operating team. Useful states might include accepted, needs review, intentionally skipped, blocked and unresolved. Keep unexpected errors separate from correct decisions to stop. An agent that refuses to invent an absent contract date should not be scored like a crashed process.

Version the event contract. If “accepted” changes from “saved successfully” to “reviewer confirmed useful,” historical trend lines are no longer comparable unless you account for that definition change.

Capture evidence references before collecting full content

For a disputed recommendation, record which source records or documents were retrieved and their relevant versions or timestamps. A URL alone may point to a page that has changed since the run. Where your data permissions allow it, retain a controlled snapshot or durable version reference for consequential evidence.

Choose what investigators genuinely need. A source identifier, retrieval timestamp, field-presence flags and validation result can answer many operational questions without duplicating the entire CRM record into the tracing platform.

Current OpenTelemetry GenAI span guidance is marked Development and lives in a separate GenAI repository. It treats instructions, input messages and outputs as potentially large and sensitive, recommends that instrumentation not capture them by default, and describes external content storage with references as an option. Pin the instrumentation and convention versions you use instead of treating old attribute examples as a permanent contract. OpenTelemetry GenAI span guidance.

LangSmith provides input and output hiding and processing options before data reaches its backend. Its examples also acknowledge that pattern-based masking can miss cases. Test the actual outgoing trace payload, including nested tool results and exception messages, rather than checking only the final answer. LangSmith sensitive-data controls.

Set retention and access according to the operational need. A short-lived debugging payload can have a different lifecycle from the durable record that a customer-facing action occurred. Redacting a visible UI field is not sufficient if the raw export still contains it.

Separate four kinds of failure

Technical failure, evidence failure, decision failure and execution failure require different remedies.

A tool timeout is a technical failure. A successful retrieval of the wrong account is an evidence or identity failure. A recommendation that contradicts current eligibility rules is a decision failure. Writing a valid recommendation to the wrong destination is an execution failure.

Track these as specific reasons rather than placing every problem in an “AI error” bucket. The fix for a slow CRM request is not necessarily a different model. The fix for a missing join key is not necessarily a longer prompt.

Also record expected stops. A suppressed contact should be skipped. A changed proposal should fail its approval check. Those outcomes can demonstrate a working control, even if a simplistic completion metric treats them as failures.

Open full diagram

This loop makes the tracing system useful to the people operating the workflow. Every recurring incident should improve either the implementation, its data contract or its evaluation coverage.

Diagnose a misleading success rate

Consider an illustrative batch of 500 account-brief tasks. The application reports 490 technically completed executions, a 98% completion rate. Review of those 490 finds thirty briefs with stale ownership, twenty with unsupported buying signals and ten saved against the wrong account.

Assume those groups do not overlap. Only 430 of the original 500 tasks produced acceptable briefs, giving an 86% acceptable-outcome rate. Reporting 98% as the agent's success rate hides sixty completed but unsuitable results.

Now classify the defects. Stale ownership points toward retrieval freshness or source selection. Unsupported buying signals require evidence and claim validation. Wrong-account writes require identity and destination checks. A global prompt rewrite could make all three groups harder to compare without addressing their causes.

Keep the population definition stable. If the next report excludes blocked tasks and missing-data cases, an apparent improvement may come from the denominator rather than a better agent. Show the number received, intentionally skipped, attempted, completed and accepted.

For delayed outcomes, preserve the observation window. A lead that has not yet received sales review is pending, not automatically rejected. Do not compare today's immature results with last month's fully reviewed cohort.

Measure time and cost at the business-task level

Report elapsed time alongside active processing time and waiting time. A task that spends twenty seconds processing and four hours waiting for review has a different bottleneck from one that repeatedly times out for four hours.

For parallel calls, adding span durations overstates wall-clock latency. Two independent six-second lookups running together can contribute roughly six seconds to the critical path, not twelve, aside from orchestration overhead. Use the trace timeline to identify the blocking path.

Aggregate cost across attempts. Suppose one illustrative task makes three model calls costing USD0.04 each and two paid data requests costing USD0.10 each. Its direct usage cost is USD0.32. A full repeated attempt with the same charges adds another USD0.32, even if the final trace screen only shows the successful attempt.

Human review and rework can dominate those amounts. Capture review time consistently and apply an explicit planning rate outside the raw event. Avoid presenting an estimated loaded labor rate as an invoice charge.

Compare cost per accepted task by release and task type. A lower average token bill is not an improvement if the new version shifts work into manual repair. Use the agent evaluation framework to test whether changes preserve useful outcomes.

Sample traces without losing the denominator

Detailed traces can be expensive, especially when tasks produce large nested outputs. Keep a reliable aggregate task ledger or counters for the overall population while choosing an appropriate policy for detailed traces.

OpenTelemetry distinguishes early sampling decisions from tail sampling, which can use information from later spans. Early sampling alone cannot guarantee retaining every eventual error. Tail sampling adds operational complexity and cannot recover trace data already discarded upstream. OpenTelemetry sampling.

For an illustrative rollout, you might retain full redacted detail for failures and a random 10% of ordinary successful tasks. That is a diagnostic sample, not the population's raw success distribution. If a dashboard divides retained failures by all retained traces, it will overstate failure prevalence.

Record the sampling policy and coverage. Late human feedback may identify a bad result after its trace has expired or been discarded. Keep enough task-level identifiers and outcome records to connect that feedback, and decide whether high-impact actions need a different retention policy.

Build an operating view people can act on

Start with a short set of questions: which release regressed, which task types fail, which source is stale, which destination writes remain uncertain, and where review time is increasing?

Assign an owner and response to each alert. An unresolved CRM write should go to someone who can reconcile the destination. A missing evidence field should reach the data or workflow owner. Repeated budget-policy rejections may belong to the product owner who defined the agent's scope.

Before relying on the dashboard:

The first useful observability release does not need dozens of charts. It needs a trustworthy path from an unacceptable business result to the evidence explaining what happened and the team able to repair it.