# Cost, Latency, and Reliability: Shipping Agents to Production
Your support agent answered the demo question perfectly. Then you turned it on for real customers, and by Friday it had run 4,000 times, cost $600, and left twelve users staring at a spinner for ninety seconds before timing out.
A demo that works once is a party trick. A product is something that still works on the hundredth run, within budget, and without hanging. This lesson covers the practical levers that get you there.
When you move an agent from "works on my laptop" to "handles real traffic," three problems show up:
Quick definitions before we go on:
Tools fail. A database is slow, an APIAPIApplication Programming Interface: a standardised interface that lets applications communicate and exchange data without knowing each other's internal workings.View full definition → returns an error, a network call stalls. Without protection, your agent waits forever or dies on the first hiccup.
Two rules fix most of this:
Timeout: set a maximum wait for every external call. If a tool does not respond in, say, 10 seconds, stop waiting and move on.
Retry: if a call fails, try again a small number of times, waiting a bit longer between attempts (this is called *exponential backoff*: wait 1s, then 2s, then 4s). Most failures are temporary and clear up on the second try.
import time
def call_tool_with_retry(tool_fn, args, retries=3, timeout=10):
for attempt in range(retries):
try:
return tool_fn(args, timeout=timeout)
except (TimeoutError, ConnectionError):
if attempt == retries - 1:
return {"error": "tool unavailable, continue without it"}
time.sleep(2 ** attempt) # 1s, 2s, 4sNotice the last line: instead of crashing, it hands the model a clear message. A good agent can often recover ("I couldn't reachreachThe number of unique people exposed to your message in a given period. Unlike impressions, reach counts each person once, no matter how often they see it.View full definition → the order system, so I asked the user for their order number instead").
For a solid, provider-neutral overview of retry patterns, see Google's SRE guidance on handling overload and retries.
An agent runs in a loop: think, call a tool, read the result, think again. The danger is a loop that never ends. The model keeps deciding "I need one more search" and you keep paying.
Always cap the number of iterations. If the agent hasn't finished in, say, 8 steps, stop and either return the best answer so far or hand off to a human.
MAX_STEPS = 8
def run_agent(user_input):
messages = [{"role": "user", "content": user_input}]
for step in range(MAX_STEPS):
response = model.respond(messages, tools=TOOLS)
if response.is_final:
return response.text
result = call_tool_with_retry(response.tool, response.args)
messages.append({"role": "tool", "content": result})
return "I couldn't resolve this fully. Escalating to a human agent."The for step in range(MAX_STEPS) line is the whole safety net. Every major agent framework (the OpenAI Agents SDK, the Claude Agent SDK, Google's ADK) has a built-in setting for this, usually called max turns or max iterations. Set it deliberately. The default is often higher than you want.
You pay for two things: what you send the model (input tokens) and what it writes back (output tokens). Output usually costs several times more per tokentokenA token is the basic unit of text that language models process, often a word fragment, whole word, or punctuation mark rather than a single character.View full definition → than input.
Three concrete ways to cut the bill:
Trim the context. Agents accumulate history: every tool result, every past message. By step 6, you might be resending 20,000 tokenstokensA token is the basic unit of text that language models process, often a word fragment, whole word, or punctuation mark rather than a single character.View full definition → of transcript on every call. Summarize old steps or drop tool results you no longer need.
Cache the stable stuff. Your system prompt and tool definitions are the same on every call. Most providers offer *prompt caching*, which charges a fraction of the price for repeated input. Turning it on is often a one-line config change and can cut input cost by 50 to 90 percent.
Ask for less output. "Answer in under 3 sentences" or returning structured data instead of prose reduces the expensive output side directly.
This is the biggest lever most teams miss. You do not need your most powerful, most expensive model for every step.
Most agent work is routine: classifying a question, extracting an order number, deciding which tool to call. A small, cheap, fast model handles these fine. Save the large model for the genuinely hard step, like writing the final nuanced reply.
This is called model routing: a cheap model does the triage, and only escalates to the expensive one when needed.
def handle(query):
# cheap model classifies
category = small_model.classify(query, ["simple_faq", "complex_issue"])
if category == "simple_faq":
return small_model.answer(query) # cheap path
else:
return large_model.run_agent(query) # expensive path, only when neededIf 70 percent of your support questions are password resets and shipping status, routing them to a small model means you only pay premium prices on the remaining 30 percent.
Knowledge check
1. According to the lesson, what distinguishes a production-ready agent from a 'party trick' demo?
2. Why does a 'chatty' agent tend to burn money quickly in production?
3. What is the purpose of exponential backoff when retrying a failed tool call?
4. Select ALL correct answers about the three problems that typically appear when moving an agent into production.
Select all the correct answers.
5. Select ALL correct answers about timeouts and retries as reliability levers.
Select all the correct answers.
Here is a real-shaped example. A support agent handles 10,000 tickets a month. The metric that matters is cost per resolution: total spend divided by tickets successfully resolved without a human.
Before (naive build):
That works out to roughly $0.42 per ticket, and about 8 percent of tickets loop until they time out, so they don't resolve at all. Some users wait 60+ seconds.
After (production build):
New numbers:
| Metric | Before | After |
|---|---|---|
| Cost per resolution | $0.42 | $0.09 |
| Median wait time | 22s | 6s |
| Tickets that hang | 8% | 0% |
Same agent, same quality of answers, roughly 4x cheaper and much faster. Nothing here required a smarter model. It required discipline about *when* to use the expensive one, and guardrails so nothing runs away.
Do not guess where the money and time go. Turn on tracing (a log of every step: which model, how many tokenstokensA token is the basic unit of text that language models process, often a word fragment, whole word, or punctuation mark rather than a single character.View full definition →, how long, success or failure). Every major agent SDK includes tracing, and open tools like Langfuse give you a vendor-neutral dashboard.
Look for the obvious wins first:
Optimize the top one or two lines. Ignore the rest until they matter.
Reliable Agents in Production
Before you point real traffic at an agent, confirm:
1. Every tool call has a timeout and a retry limit.
2. The agent loop has a hard iteration cap with a clean fallback.
3. Prompt caching is on, and context gets trimmed.
4. Routine steps use a smaller model.
5. Tracing is on so you can see cost and latency per run.
Provider-specific details (exact setting names, how to enable caching) live in the deep-dive blocks for OpenAI, Claude, and Gemini. The levers above transfer across all of them.