Cost, latency, and reliability: shipping agents to production

# Cost, latency, and reliability: shipping agents to production

Your support agent answered the demo question perfectly. Then you turned it on for real customers, and by Friday it had run 4,000 times, cost $600, and left twelve users staring at a spinner for ninety seconds before timing out.

A demo that works once is a party trick. A product is something that still works on the hundredth run, within budget, and without hanging. This lesson covers the practical levers that get you there.

The three things that break in production

When you move an agent from "works on my laptop" to "handles real traffic," three problems show up:

• Cost: every step an agent takes sends text to a model, and you pay per unit of text. A chatty agent burns money fast.
• Latency: how long the user waits. Agents that "think" across many steps are slow.
• Reliability: does it finish, or does it hang, loop forever, or crash on a bad tool response?

Quick definitions before we go on:

• Agent: a program that uses a large language modellarge language modelA Large Language Model is an AI system trained on vast text data to predict and generate language, enabling tasks like writing, summarizing, and answering questions.View full definition → to decide what to do next, often calling tools (a search, a database lookup, an email send) in a loop until the task is done.
• TokenTokenA token is the basic unit of text that language models process, often a word fragment, whole word, or punctuation mark rather than a single character.View full definition →: the unit models charge by. Roughly 3 to 4 characters, or about 0.75 words. "Reset my password" is about 4 tokenstokensA token is the basic unit of text that language models process, often a word fragment, whole word, or punctuation mark rather than a single character.View full definition →.

Lever 1: Timeouts and retries

Tools fail. A database is slow, an APIAPIApplication Programming Interface: a standardised interface that lets applications communicate and exchange data without knowing each other's internal workings.View full definition → returns an error, a network call stalls. Without protection, your agent waits forever or dies on the first hiccup.

Two rules fix most of this:

Timeout: set a maximum wait for every external call. If a tool does not respond in, say, 10 seconds, stop waiting and move on.

Retry: if a call fails, try again a small number of times, waiting a bit longer between attempts (this is called *exponential backoff*: wait 1s, then 2s, then 4s). Most failures are temporary and clear up on the second try.

import time

def call_tool_with_retry(tool_fn, args, retries=3, timeout=10):
    for attempt in range(retries):
        try:
            return tool_fn(args, timeout=timeout)
        except (TimeoutError, ConnectionError):
            if attempt == retries - 1:
                return {"error": "tool unavailable, continue without it"}
            time.sleep(2 ** attempt)  # 1s, 2s, 4s

Notice the last line: instead of crashing, it hands the model a clear message. A good agent can often recover ("I couldn't reachreachThe number of unique people exposed to your message in a given period. Unlike impressions, reach counts each person once, no matter how often they see it.View full definition → the order system, so I asked the user for their order number instead").

For a solid, provider-neutral overview of retry patterns, see Google's SRE guidance on handling overload and retries.

Lever 2: Cap the loop

An agent runs in a loop: think, call a tool, read the result, think again. The danger is a loop that never ends. The model keeps deciding "I need one more search" and you keep paying.

Always cap the number of iterations. If the agent hasn't finished in, say, 8 steps, stop and either return the best answer so far or hand off to a human.

MAX_STEPS = 8

def run_agent(user_input):
    messages = [{"role": "user", "content": user_input}]
    for step in range(MAX_STEPS):
        response = model.respond(messages, tools=TOOLS)
        if response.is_final:
            return response.text
        result = call_tool_with_retry(response.tool, response.args)
        messages.append({"role": "tool", "content": result})
    return "I couldn't resolve this fully. Escalating to a human agent."

The for step in range(MAX_STEPS) line is the whole safety net. Every major agent framework (the OpenAI Agents SDK, the Claude Agent SDK, Google's ADK) has a built-in setting for this, usually called max turns or max iterations. Set it deliberately. The default is often higher than you want.

Lever 3: Control tokentokenA token is the basic unit of text that language models process, often a word fragment, whole word, or punctuation mark rather than a single character.View full definition → cost

You pay for two things: what you send the model (input tokens) and what it writes back (output tokens). Output usually costs several times more per tokentokenA token is the basic unit of text that language models process, often a word fragment, whole word, or punctuation mark rather than a single character.View full definition → than input.

Three concrete ways to cut the bill:

Trim the context. Agents accumulate history: every tool result, every past message. By step 6, you might be resending 20,000 tokenstokensA token is the basic unit of text that language models process, often a word fragment, whole word, or punctuation mark rather than a single character.View full definition → of transcript on every call. Summarize old steps or drop tool results you no longer need.

Cache the stable stuff. Your system prompt and tool definitions are the same on every call. Most providers offer *prompt caching*, which charges a fraction of the price for repeated input. Turning it on is often a one-line config change and can cut input cost by 50 to 90 percent.

Ask for less output. "Answer in under 3 sentences" or returning structured data instead of prose reduces the expensive output side directly.

Lever 4: Use a smaller model for routine steps

This is the biggest lever most teams miss. You do not need your most powerful, most expensive model for every step.

Most agent work is routine: classifying a question, extracting an order number, deciding which tool to call. A small, cheap, fast model handles these fine. Save the large model for the genuinely hard step, like writing the final nuanced reply.

This is called model routing: a cheap model does the triage, and only escalates to the expensive one when needed.

def handle(query):
    # cheap model classifies
    category = small_model.classify(query, ["simple_faq", "complex_issue"])

    if category == "simple_faq":
        return small_model.answer(query)      # cheap path
    else:
        return large_model.run_agent(query)   # expensive path, only when needed

If 70 percent of your support questions are password resets and shipping status, routing them to a small model means you only pay premium prices on the remaining 30 percent.

Before and after: a support agent's cost per resolution

Here is a real-shaped example. A support agent handles 10,000 tickets a month. The metric that matters is cost per resolution: total spend divided by tickets successfully resolved without a human.

Before (naive build):

• Every ticket uses the large model for all steps.
• Full conversation history resent every step. No caching.
• No iteration cap, so stuck tickets loop up to 20 times.
• Average: 15 model calls per ticket, ~18,000 tokenstokensA token is the basic unit of text that language models process, often a word fragment, whole word, or punctuation mark rather than a single character.View full definition → each.

That works out to roughly $0.42 per ticket, and about 8 percent of tickets loop until they time out, so they don't resolve at all. Some users wait 60+ seconds.

After (production build):

• A small model triages. 65 percent of tickets (FAQs, status checks) resolve on the cheap path.
• Prompt caching on the system prompt and tools.
• Context trimmed: old tool results summarized after 3 steps.
• Iteration cap at 8, with clean escalation to a human.

New numbers:

| Metric | Before | After |

|---|---|---|

| Cost per resolution | $0.42 | $0.09 |

| Median wait time | 22s | 6s |

| Tickets that hang | 8% | 0% |

Same agent, same quality of answers, roughly 4x cheaper and much faster. Nothing here required a smarter model. It required discipline about *when* to use the expensive one, and guardrails so nothing runs away.

Measure before you optimize

Do not guess where the money and time go. Turn on tracing (a log of every step: which model, how many tokenstokensA token is the basic unit of text that language models process, often a word fragment, whole word, or punctuation mark rather than a single character.View full definition →, how long, success or failure). Every major agent SDK includes tracing, and open tools like Langfuse give you a vendor-neutral dashboard.

Look for the obvious wins first:

• Which step uses the most tokenstokensA token is the basic unit of text that language models process, often a word fragment, whole word, or punctuation mark rather than a single character.View full definition →? (Usually a bloated context.)
• Which tool is slowest? (Add a timeout.)
• How many steps does a typical ticket take? (If it's near your cap, investigate.)

Optimize the top one or two lines. Ignore the rest until they matter.

A simple shipping checklist

Before you point real traffic at an agent, confirm:

1. Every tool call has a timeout and a retry limit.

2. The agent loop has a hard iteration cap with a clean fallback.

3. Prompt caching is on, and context gets trimmed.

4. Routine steps use a smaller model.

5. Tracing is on so you can see cost and latency per run.

Provider-specific details (exact setting names, how to enable caching) live in the deep-dive blocks for OpenAI, Claude, and Gemini. The levers above transfer across all of them.

Key Takeaways

• Cap the loop and set timeouts on every tool. These two guardrails prevent the two worst production failures: runaway cost and hanging requests.
• Route by difficulty. Use a cheap, fast model for triage and routine steps; reserve the expensive model for the genuinely hard work. This is usually your largest cost win.
• Track cost per resolution, not cost per call. It ties spend to actual value delivered and exposes tickets that burn money without resolving.
• Trim context and turn on prompt caching. Agents quietly resend growing histories; both moves cut input cost sharply with minimal effort.
• Turn on tracing before optimizing. Measure which step is slow or expensive, fix the top one or two, and stop there.