# Evaluating and Debugging Agents: Traces, Evals, and Failure Modes
A customer asks your travel agent bot to "book the cheapest flight to Boston next Friday," and it confidently books a flight to Boston, Lincolnshire, in the UK. The customer wanted Boston, Massachusetts. The answer looks fine until you see what the agent actually did: it never called the tool that resolves airport codes, so it guessed.
That gap between "looks fine" and "what actually happened" is where debugging agents lives. You cannot fix what you cannot see.
An agent is an AI system that can take actions, not just chat. It calls tools (functions like "search flights," "read a file," or "send an email"), reads the results, and decides what to do next.
A trace is the recorded step-by-step log of one run: every prompt the model saw, every tool it called, the arguments it passed, what came back, and its final answer. Think of it as a flight recorder for a single agent task.
Without a trace, debugging is guessing. With one, the failure usually jumps out.
Here is a simplified trace of the Boston booking, written in plain language:
RUN #4471 | user: "book the cheapest flight to Boston next Friday"
Step 1 LLM thinks: "User wants a flight. I'll search."
Step 2 TOOL CALL: search_flights(city="Boston", date="2026-01-16")
Step 3 TOOL RESULT: 3 flights returned [destination code: BZZ, UK]
Step 4 LLM thinks: "Cheapest is £48. Booking it."
Step 5 TOOL CALL: book_flight(flight_id="BZZ-2231")
Step 6 TOOL RESULT: {"status": "confirmed"}
Step 7 LLM final: "Booked! £48 to Boston next Friday."The bug is in Step 2. The agent passed a plain city name and never disambiguated it. There is no "resolve_airport" step. The model did not fail at reasoning; it was missing a step in the workflow.
That is the point of tracing: it tells you *where* things broke, so you know *what* to fix.
When you open a trace of a bad run, scan for these:
city="Boston" with no country.Most agent failures are one of these five. You do not need to read code to spot them; you read the trace like a story and find the sentence that stops making sense.
You do not build a trace viewer yourself. Modern agent frameworks emit traces automatically, and observability tools display them as clickable timelines. Open, free options include Langfuse and Arize Phoenix, both of which support the OpenTelemetry standard so they work across providers.
Each vendor's agent SDK (the OpenAI Agents SDK, the Claude Agent SDK, Google's ADK) also has built-in tracing. The concept is identical everywhere; only the buttons differ. See the provider deep-dive blocks for the exact setup.
Here is what emitting a trace looks like in vendor-neutral pseudocode. Notice the loop is just: think, call tool, feed result back, repeat.
trace = []
while not done:
response = model.generate(messages, tools=my_tools)
trace.append({"step": "llm", "output": response})
if response.tool_call:
result = run_tool(response.tool_call)
trace.append({"step": "tool",
"name": response.tool_call.name,
"args": response.tool_call.args,
"result": result})
messages.append(result) # feed result back to the model
else:
done = True
save(trace) # now you can inspect this run laterThe trace.append lines are the whole trick. Log every step as it happens, and you can replay any run afterward.
Once you have read a few dozen traces, patterns repeat. Here are the big ones and the usual fix.
The agent says "I checked the database" but the trace shows no tool call. Fix: strengthen the instruction ("You must call lookup_order before answering about any order") and, if the framework allows, force a tool call for that step.
The tool expects a date like 2026-01-16 but the agent sends "next Friday." Fix: describe the argument format clearly in the tool's description, and add a validation step that rejects bad input and asks the model to retry.
On a 15-step task, the agent forgets what it was doing by step 10. Fix: break the task into smaller sub-tasks, or add a short "current goal" reminder to each step.
The agent calls a web search five times for one question, burning time and money. Fix: set a step limit and instruct it to summarize what it already knows before searching again.
The fix is almost always one of three things: a clearer tool description, a clearer instruction, or a guardrail (a limit or a validation check). You rarely need a bigger model.
You found the Boston bug. You added a "resolve_airport" step. Did it work? And did it break anything else?
This is where evals come in. An eval (short for evaluation) is a repeatable test of your agent against a set of example inputs with known good outcomes. It is the agent equivalent of a checklist a pilot runs before every flight.
The core idea is an eval set: a small collection of test cases you run every time you change the agent. Each case has an input and a way to check the output.
Start with a simple spreadsheet or file. Ten to twenty cases is plenty to begin.
case_id | input | check
--------|------------------------------------------|----------------------------
1 | "cheapest flight to Boston next Friday" | destination country == "US"
2 | "flight to Paris" | asks which Paris OR picks FR
3 | "cancel my last booking" | calls cancel_booking exactly once
4 | "book London to Boston" | resolves BOTH cities
5 | "what's the weather" | does NOT call book_flightNotice the checks are concrete. Some are exact ("destination country == US"), and some are behavioral ("does not call book_flight"). Case 5 is a negative test: it makes sure the agent does *not* do something wrong.
The loop is simple:
1. Run all cases against the current agent. Record pass/fail.
2. Make one change (add the airport step).
3. Run all cases again.
4. Compare. Did case 1 flip from fail to pass? Did any case flip from pass to fail?
That last question catches regressions: a fix that quietly breaks something that used to work. Without an eval set, you would ship the fix and find out from an angry customer.
Many agent outputs are not exact strings. "Sure, I've cancelled your booking" and "Done, your booking is cancelled" both pass. For these, two common approaches:
cancel_booking once. This is more reliable than checking the wording.Prefer behavioral checks when you can. They are cheaper and far more stable than judging text.
Vérification des acquis
1. According to the lesson, what is an agent trace?
2. In the Boston booking example, why did the agent fail?
3. What is the core insight behind the phrase 'You cannot fix what you cannot see'?
4. Select ALL correct answers. What does a trace typically capture for a single agent run?
Sélectionnez toutes les réponses correctes.
5. Select ALL correct answers. Which statements reflect the lesson's distinction between an agent and a plain chatbot?
Sélectionnez toutes les réponses correctes.
Here is the whole workflow in order. This is the habit to build.
1. A run fails. A user reports a bad answer, or an eval case fails.
2. Open the trace. Read it as a story. Find the step that stops making sense.
3. Name the failure mode. Wrong tool? Bad arguments? Ignored result? Loop?
4. Make one change. Clearer instruction, clearer tool description, or a guardrail.
5. Run the eval set. Confirm the target case passes and nothing else regressed.
6. Add the failing case to your eval set so it never regresses silently again.
Step 6 is the one people skip, and it is the most valuable. Every bug you fix becomes a permanent test. Over a few months your eval set grows into a real safety net, built entirely from real failures.
One change at a time matters too. If you fix the airport step *and* rewrite the instructions *and* swap the model all at once, and the score moves, you will not know which change did it. Change one thing, run the evals, repeat.