When an AI agent produces a bad answer, the visible sentence is often the last link in a longer failure. The wrong source may have been retrieved, a tool may have received an ambiguous argument, a handoff may have lost a constraint, or a guardrail may have rejected a call and left the workflow with an empty result. Rewriting the final prompt can appear to help while leaving the original break intact.
The practical move is to trace each meaningful boundary in one agent run, then find the first broken span: the earliest operation whose recorded input, output, or status no longer supports the next operation. A span is simply a timed record of one step inside an end-to-end trace. Think of it as a receipt for “retrieved documents,” “called inventory lookup,” or “drafted answer,” connected to the same request.
OpenAI’s Agents SDK describes a trace as one end-to-end workflow operation made of spans, and documents spans for agent runs, model generations, function calls, guardrails, and handoffs.[1] Anthropic’s current cookbook includes recipes for tracing API requests and agent sessions with OpenTelemetry, as well as reviewing finished sessions for problems with quoted evidence.[2] The useful lesson is not to collect every possible log field. It is to make the workflow’s decision boundaries inspectable enough to identify where a good chain first becomes unsupported.
Follow the evidence, not the final prose
Start with a single failed or suspicious run. Put the user-visible outcome at the end of a small chain:
- request received
- scope or policy selected
- retrieval or tool choice made
- tool arguments sent
- tool result returned
- model used the result
- answer, proposal, or action produced
At each boundary, ask one narrow question: does this record justify the next step? If the answer cited an outdated policy, do not begin by criticizing its wording. First inspect whether the trace retrieved the outdated document, ranked it above the current version, sent both versions to the model, or failed to preserve the version filter. The first “no” is the repair target.
This prevents a common debugging loop: changing the final instruction, seeing a temporary improvement on one example, and making the next failure harder to explain. A trace separates an evidence problem from a selection problem and a generation problem.
Copy this first-broken-span worksheet
Use this after a real miss. It works in a tracing dashboard, a log explorer, or a simple table. Keep user and sensitive data out of fields that do not need it.
RUN
- trace_id: [stable ID]
- expected outcome: [observable result]
- actual outcome: [what happened]
- request class: [e.g., policy lookup / support routing]
SPAN CHAIN
1. [span name / type]
input boundary: [what it was allowed to receive]
observed result: [ID, status, count, or short safe excerpt]
expected evidence: [what had to be true for the next span]
supports next span? yes | no | unknown
2. [repeat for each meaningful step]
FIRST BREAK
- earliest span with no/unknown: [span ID and name]
- failure class: missing input | bad selection | invalid tool args |
tool failure | policy/guardrail block | unsupported generation
- evidence: [specific field, status, or linked record]
- smallest repair hypothesis: [one change to test]
VERIFY
- rerun on: [original case + one nearby boundary case]
- success means: [changed trace condition and user-visible outcome]
- do not change yet: [later prompts, unrelated tools, model version]
The expected evidence line is the important part. “Retrieval completed” is not enough. A retrieval span for a version-specific question should preserve the query, filters, returned document IDs, and versions. A tool-call span should preserve the allowed operation and validated arguments, not just “HTTP 200.” An answer span should let you see which returned facts it was asked to use. Record stable IDs and counts when full content would be sensitive.
Instrument boundaries with purpose
Do not make one giant span called agent_run and hope it tells a story. Name the operations where a decision can change the outcome: retrieve_policy, rank_candidates, get_account, check_permission, draft_reply, request_approval, and publish. Add a trace ID that connects them, a parent relationship that shows order, and small attributes that explain the boundary: corpus version, selected document IDs, tool name, result status, model or prompt version, and whether a guardrail passed.
OpenAI cautions that generation and function spans can contain sensitive inputs and outputs; its tracing guide documents a setting to omit that data.[1] Treat observability as a data-design task, not permission to export transcripts. Prefer IDs, safe categories, counts, and explicit redaction. If a complete payload is truly needed for a secure debugging workflow, restrict access and retention separately.
Failure mode: tracing everything and learning nothing
A trace can become a pile of verbose events that no one reads. It can also expose private customer data while still failing to record the one metadata field—such as document version or approval result—that explains a mistake. Start from the questions operators need to answer: Which source won? Which tool received which validated target? Where did a guardrail block progress? Add fields that answer those questions, then sample real failed traces to confirm they do.
The habit to keep
when an agent misses, trace backward from the bad outcome and repair the first boundary that no longer supports the next one. Better final prose is valuable, but it cannot reliably repair evidence that was never retrieved or a tool call that was wrong before the model wrote a word.