AI Agent Observability: Tracing Decisions in Production

Something goes wrong with your AI agent in production. A customer reports that it took a bad action: sent the wrong email, skipped a required validation step, called an API with malformed parameters. You open your logs. You see the final output. You see that the action happened. You have no idea why.

That's the default state for most agent deployments today, and it's a serious reliability problem. Traditional application logging captures what your system did. Agent observability requires capturing why: the full reasoning trace, every tool call with its exact input and output, and the model's intermediate steps. Without that, debugging is pure speculation. You're reconstructing a crime scene from a single footprint.

LLM observability for agents isn't a monitoring dashboard you bolt on after the fact. It's an instrumentation discipline you design into the agent from the start. The Laxaar team has instrumented agents across customer support, operations automation, and document processing pipelines. The pattern holds every time: teams that skip tracing regret it the first time something breaks in a way they can't reproduce.

What you'll learn

Why agent debugging fails without traces
The anatomy of an agent trace
Instrumenting an agent with OpenTelemetry spans
What to capture in each span
Choosing an LLM observability platform
Attaching evals to traces for quality signals
Common instrumentation mistakes
Frequently Asked Questions

Why agent debugging fails without traces

A web server logs requests and responses. When something breaks, you find the request, read the response, and usually have enough context to understand the failure. Agents don't work that way.

An agent run might span dozens of LLM calls, five tool invocations, three memory reads, and a branching decision tree, all in service of answering one user query or completing one task. The final output is a single artifact that tells you almost nothing about the path that produced it. A wrong answer could have come from a bad retrieval, a misinterpreted tool result, a malformed intermediate prompt, or a model reasoning error at step six of twelve. Output logging can't distinguish between these.

There's also the non-determinism problem. Ask an agent the same question twice and you may get meaningfully different behavior due to temperature, tool result ordering, or in-context learning from earlier turns. You can't reproduce a failure by replaying inputs. You need the actual execution trace from the failing run, captured at the time it happened.

The standard for production-grade agent observability is treating each agent run as a distributed trace: a tree of spans where each span represents one unit of work (one LLM call, one tool invocation, one memory lookup) with timing, inputs, outputs, and metadata attached.

The anatomy of an agent trace

An agent trace is a hierarchical structure rooted at the top-level agent invocation, with child spans for every sub-operation the agent performs. The shape looks like this:

Root span. The full agent run. Contains the user input, final output, total latency, total token usage, and run metadata (model, agent version, session ID).
LLM call spans. One span per model invocation. Contains the exact prompt sent (system prompt plus assembled context), the raw completion, token counts, latency, and the model/temperature used.
Tool call spans. One span per tool invocation. Contains the tool name, the exact arguments the model generated, the raw return value, and any error raised. This is the most important span for debugging action failures.
Memory spans. Reads from and writes to any external memory store. Contains the query, the results returned, and retrieval latency.
Planner spans. If your agent uses a planner/executor pattern, a span wrapping the planning step so you can see what plan was generated and how it was executed.

Tool call spans, not LLM call spans, are where most production failures live. The model reasoned correctly but called the tool with arguments it interpreted from ambiguous context. The tool returned an error the agent didn't handle. A downstream API returned a 429 and the agent silently skipped the action. None of this is visible in output logs. All of it is visible in a properly structured tool call span.

Instrumenting an agent with OpenTelemetry spans

OpenTelemetry is the right foundation for agent tracing. It's vendor-neutral, ships SDKs for every language, and integrates with every major observability platform. You don't need an LLM-specific tracing library to start: wrap your agent operations in OTel spans and export to whatever backend you're using.

from opentelemetry import trace
from opentelemetry.sdk.trace import TracerProvider
from opentelemetry.sdk.trace.export import BatchSpanProcessor
from opentelemetry.exporter.otlp.proto.http.trace_exporter import OTLPSpanExporter
import time

# Initialize the tracer
provider = TracerProvider()
provider.add_span_processor(
    BatchSpanProcessor(OTLPSpanExporter(endpoint="http://localhost:4318/v1/traces"))
)
trace.set_tracer_provider(provider)
tracer = trace.get_tracer("agent.tracer")


def run_agent(user_input: str, session_id: str) -> str:
    with tracer.start_as_current_span("agent.run") as root_span:
        root_span.set_attribute("session.id", session_id)
        root_span.set_attribute("agent.input", user_input)
        root_span.set_attribute("agent.version", "1.4.2")

        # Step 1: retrieve context from memory
        context = retrieve_context(user_input)

        # Step 2: call the LLM
        plan = call_llm(user_input, context)

        # Step 3: execute tools from the plan
        result = execute_tools(plan)

        root_span.set_attribute("agent.output", result)
        root_span.set_attribute("agent.total_tokens", root_span.attributes.get("llm.total_tokens", 0))
        return result


def call_llm(prompt: str, context: str) -> str:
    with tracer.start_as_current_span("llm.call") as span:
        span.set_attribute("llm.model", "gpt-4o")
        span.set_attribute("llm.prompt", prompt[:2000])   # truncate for span size
        span.set_attribute("llm.context_length", len(context))

        start = time.time()
        response = your_llm_client.complete(prompt=prompt, context=context)
        span.set_attribute("llm.latency_ms", int((time.time() - start) * 1000))
        span.set_attribute("llm.prompt_tokens", response.usage.prompt_tokens)
        span.set_attribute("llm.completion_tokens", response.usage.completion_tokens)
        span.set_attribute("llm.output", response.content[:2000])
        return response.content


def execute_tool(tool_name: str, tool_args: dict) -> str:
    with tracer.start_as_current_span("tool.call") as span:
        span.set_attribute("tool.name", tool_name)
        span.set_attribute("tool.args", str(tool_args))
        try:
            result = dispatch_tool(tool_name, tool_args)
            span.set_attribute("tool.result", str(result)[:2000])
            span.set_attribute("tool.success", True)
            return result
        except Exception as e:
            span.set_attribute("tool.success", False)
            span.set_attribute("tool.error", str(e))
            span.record_exception(e)
            raise

This is the minimum viable instrumentation. Every LLM call and tool dispatch has a span. Each span carries enough context to reconstruct the decision. You can export to Langfuse, Arize Phoenix, or any OTLP-compatible backend without changing the instrumentation code.

What to capture in each span

Span attribute selection is where teams make consequential trade-offs. Capture too little and the trace is useless for debugging. Capture too much and you hit span size limits, inflate storage costs, and risk leaking sensitive data.

Span Type	Must Capture	Skip or Truncate
LLM call	Model ID, prompt (truncated), completion (truncated), token counts, latency, temperature	Full system prompt if it's static — log it once at startup instead
Tool call	Tool name, exact args, raw return value, success/failure, error message	Internal retry state, raw HTTP response bodies over 4KB
Memory read	Query text, number of results, top result score (for vector), latency	Full text of all retrieved documents — attach by reference if needed
Root span	Session ID, user ID (hashed), agent version, total tokens, final output	Raw PII in user input — hash or redact before attaching

A practical rule: capture anything you'd need to reproduce the failure, redact anything that would violate a data protection obligation, and truncate anything that's long but not decision-relevant. Prompts are decision-relevant. A 50KB retrieved document is not. A summary or a reference ID is enough.

Choosing an LLM observability platform

You have three categories of tooling to consider for receiving and querying agent traces:

Purpose-built LLM observability tools. Langfuse, Helicone, and Arize Phoenix are the current leaders. All three accept OTLP traces or have native SDKs, display traces as readable conversation threads with tool call expansions, and support cost attribution per run. Langfuse is open-source and self-hostable, which matters if you can't send model I/O to a third-party SaaS. Arize Phoenix has the strongest eval integration. Helicone is the lightest integration path if you're calling OpenAI-compatible endpoints directly.

General-purpose APM with LLM plugins. Datadog, Grafana, and New Relic all have LLM observability add-ons. If you're already running on one of these platforms and your team is fluent in it, the add-on is the path of least resistance. The trade-off is that LLM-specific views (conversation thread rendering, token cost attribution, prompt diff tools) are less mature than purpose-built tools.

Self-managed OTLP stack. Jaeger or Tempo for trace storage, Prometheus for metrics, Grafana for dashboards. Maximum control, no third-party data sharing, higher operational cost. Reasonable for compliance-sensitive deployments.

For most teams building production AI agents, the right starting point is Langfuse (self-hosted if data sovereignty matters, cloud otherwise) paired with OpenTelemetry instrumentation. You get conversation-thread trace views, cost attribution, and an eval pipeline, all without locking into a proprietary SDK.

Attaching evals to traces for quality signals

Tracing tells you what happened. Evals tell you whether what happened was correct. The two belong together: an eval score attached to a trace span turns a debugging artifact into a quality signal you can track over time.

The minimal approach is attaching a pass/fail eval to the root span at run time. A rule-based eval (did the agent call the right tool? did the output match the expected schema?) runs in milliseconds and adds a eval.correct boolean to the span. When that boolean flips to false, you have a trace to inspect.

def evaluate_run(trace_id: str, agent_output: str, expected_tool: str, tools_called: list[str]) -> dict:
    correct_tool = expected_tool in tools_called
    output_valid = validate_output_schema(agent_output)  # your schema check

    eval_result = {
        "eval.correct_tool": correct_tool,
        "eval.output_valid": output_valid,
        "eval.pass": correct_tool and output_valid,
    }

    # Attach eval results back to the root span
    with tracer.start_as_current_span("eval.run") as span:
        span.set_attribute("eval.trace_id", trace_id)
        for key, val in eval_result.items():
            span.set_attribute(key, val)

    return eval_result

LLM-judge evals (where a separate model scores the output) are more expensive but catch quality failures that rule-based checks miss. Run these asynchronously after the agent run completes, attach the score to the trace via the trace ID, and build an alert on sliding-window pass rate. A drop from 94% to 87% over 200 runs is a signal worth investigating before a user files a ticket.

The Laxaar team typically recommends a three-tier eval stack: fast rule-based checks in-line, LLM-judge evals async within seconds, and human review for low-confidence or flagged traces. All three feed back into the same trace. Debugging and quality workflows share a single artifact rather than two disconnected systems.

Common instrumentation mistakes

Logging outputs instead of traces. Storing agent_output: "I've sent the email" in a log table is not observability. It tells you the agent claimed to do something. A trace with a tool call span tells you whether the send_email tool was called, with what arguments, and whether it returned success. These are different facts.

Losing the trace on retries. If your agent retries a failed tool call, each retry needs to be a child span of the original tool call span, not a new top-level trace. Retry loops that overwrite the root span ID make it impossible to see that a retry happened and what changed between attempts.

No span for the planning step. Teams instrument LLM calls and tool calls but skip the planner. When the agent takes a wrong action path, the failure is usually in the plan, not in the execution. If the planner span isn't captured, you can see the wrong action was taken but not why the agent decided to take it.

Sampling too aggressively. Standard APM sampling at 1% or 5% is catastrophic for agent debugging. An agent making 200 runs per day at 1% sampling means you have traces for two runs. When one of those runs fails, there's a high probability you don't have the trace. Sample at 100% for agents in development and staging. In production, sample 100% of failed runs (use tail-based sampling) and a meaningful fraction, 20% or more, of successful ones.

Redacting too much. Some teams, correctly worried about PII, redact so aggressively that the tool call args are fully masked and the LLM prompt is a blank field. The trace becomes useless for debugging while still containing enough metadata to raise compliance concerns anyway. Apply targeted redaction: mask actual PII fields (email addresses, phone numbers, SSNs) and preserve structure and non-sensitive context.

At Laxaar, instrumentation is a first-class engineering deliverable on every agent project, not a logging afterthought bolted on after the first production incident. For teams building out custom software development stacks that include agent components, that discipline pays back quickly.

Frequently Asked Questions

How is agent tracing different from standard application tracing?

Standard distributed tracing tracks request flow across services: a span per HTTP call, a span per database query. Agent tracing adds spans for non-deterministic, model-driven decisions. Each LLM call produces different output for the same input, tool calls are chosen by the model rather than deterministic code paths, and reasoning steps are implicit rather than explicit in the code. This means agent traces need to capture the model's reasoning artifacts (prompts, completions, tool schemas), not just latency and status codes. The tooling is similar (OpenTelemetry works fine), but the data model is richer.

What's a reasonable storage cost for full agent traces?

A typical agent run with five LLM calls and three tool invocations, fully instrumented with prompts and completions, generates roughly 20-80KB of span data depending on context size. At 10,000 runs per day, that's 200MB to 800MB of trace data daily. With a 30-day retention window, you're looking at 6-24GB, manageable in any modern trace backend. If cost is a concern, apply retention tiers: keep full traces for 7 days, downsample to root spans only after that, and retain root spans for 90 days for trend analysis.

Can we add observability to an agent we didn't build ourselves?

Yes, with limitations. If you're using an agent framework (LangGraph, CrewAI, AutoGen), check whether it exposes OpenTelemetry instrumentation hooks; most modern frameworks do. You can add spans at the framework callback level without modifying the agent logic itself. If you're calling a third-party agent API, you can wrap the API call in a span and capture inputs and outputs, but you won't have visibility into the agent's internal reasoning steps. In that case, push the vendor to expose a trace export endpoint. It's a reasonable ask for any production deployment.

Should we trace every run, or only failures?

Trace every run in development and staging without exception. In production, capture every run but consider async trace flushing to avoid blocking agent latency. For storage, use tail-based sampling that guarantees full retention for any run that results in an error, a low eval score, or user-reported feedback, and a configurable sampling rate (20-50%) for successful, high-confidence runs. The worst time to wish you had a trace is during a production incident for a run that got sampled away.

How do we handle traces for long-running or multi-day agent tasks?

Long-running agents (those that run for hours or days across multiple sessions) should use a persistent trace context rather than a single open span. The pattern is a root trace with checkpointed child spans: each work segment (one planning cycle, one batch of tool calls) creates a child span that completes and flushes independently. The root trace stays open until the task completes or is explicitly closed. Langfuse and Arize both support this pattern with their session and trace threading APIs. Avoid keeping a single OTLP span open for hours; most backends enforce a span duration limit and will drop it.

Building an AI agent that needs production-grade observability? The Laxaar team designs and instruments agent systems with tracing, evals, and alerting built in from day one. Reach out and we'll help you avoid the debugging pain that comes from shipping without traces.