+150 XP

Cost, latency, and model selection tradeoffs

A startup ran 2 million customer reviews through a frontier reasoning model (GPT-5 at default settings) to sort them into "positive," "negative," or "neutral." It worked great. It also took 9 hours and cost about $1,500, most of it spent on hidden "thinking" tokens nobody needed. They switched to a small, fast model with reasoning turned down. Same job: 40 minutes, about $40, and accuracy dropped by less than 1%.

That is the whole lesson in one story. The biggest model is rarely the right one for high-volume, repetitive work. Here is how to think about it.

Three things you're always trading

Every time you pick a model, you balance three things:

  • Quality: how good the output is.
  • Latency: how long you wait for a response. (Latency just means delay.)
  • Cost: how much you pay per request.

You can usually optimize for two, but not all three. A huge frontier model gives you top quality but costs more and runs slower. A small model is cheap and fast but may stumble on hard tasks.

The skill is matching the model to the job, not always reaching for the most powerful one.

Why bigger models are slower and pricier

Large language models charge by the token, a token being roughly 3/4 of a word. "Customer service" is about 2 to 3 tokens.

Bigger models have more internal "parameters" (the dials they tune during training). More parameters means more computation per token, which means more cost and more delay.

There is a second cost that is easy to miss. Many current models (GPT-5, Claude with extended thinking, Gemini 2.5 Pro) are reasoning models: before answering, they generate hidden reasoning tokens. You never see them, but you pay for them at the output rate and you wait for them. On a one-word answer, a model might "think" for hundreds of tokens first.

So a frontier model can easily cost 10x to 50x more per request than a small one and respond several times slower. That difference is invisible on one request. On 2 million requests, it is your whole budget.

A concrete example: the classification job

Let's make the review-sorting task real.

You have 2 million reviews. Each is short (say 50 tokens in, one word out, just the label). You want each tagged positive, negative, or neutral. This is classification: sorting items into fixed buckets.

Here is the prompt you'd send for each review:

Classify the sentiment of this review as exactly one word:
positive, negative, or neutral.

Review: "Shipping was slow but the product is excellent."

Answer:

This is an easy task. Even small models handle sentiment well. You do not need a model that can write poetry or debug code. You need one that reads three sentences and outputs one word, fast and cheap.

Run the numbers

Rough late-2026 pricing (always check current rates, they move often):

Model typeCost per 1M input tokensCost per 1M output tokensRelative speed
Frontier (GPT-5, Claude Opus, Gemini Pro)~$1.25 to $5~$10 to $25Slower
Small/fast (GPT-5 mini or nano, Claude Haiku, Gemini Flash)~$0.05 to $1~$0.40 to $5Much faster

For 2 million reviews at ~50 tokens each, that's 100 million input tokens.

  • Frontier model: roughly $125 to $500 for input alone. Add a few hundred hidden reasoning tokens per review at output prices and the bill climbs into the thousands, plus many hours.
  • Small model with reasoning kept minimal: roughly $5 to $100, in a fraction of the time.

The small model wins decisively, as long as accuracy holds. For a task this simple, it usually does. That is the catch you must verify, which brings us to testing.

How to actually decide: test, don't guess

Never assume the small model is good enough. Measure it. Here's the workflow.

Step 1: Build a small labeled test set

Hand-label 100 to 200 reviews yourself. This is your ground truth: the correct answers you compare against.

Step 2: Run both models on the test set

python
from openai import OpenAI
client = OpenAI()

def classify(review, model):
    resp = client.chat.completions.create(
        model=model,
        messages=[{
            "role": "user",
            "content": f"Classify sentiment as one word "
                       f"(positive, negative, neutral): {review}"
        }],
        reasoning_effort="minimal",   # skip most hidden thinking
        max_completion_tokens=16,     # includes reasoning tokens
    )
    return (resp.choices[0].message.content or "").strip().lower()

# Compare a small model vs a large one
for model in ["gpt-5-mini", "gpt-5"]:
    correct = sum(
        classify(r["text"], model) == r["label"]
        for r in test_set
    )
    print(model, correct / len(test_set))

Two details matter here. Reasoning models reject the old max_tokens parameter, so you use max_completion_tokens. And that cap counts hidden reasoning tokens too. Set it to 1 and the model may spend its single token thinking and return nothing at all.

Step 3: Compare accuracy and decide

Say the big model scores 96% and the small one scores 95%. That 1% gap, on a review-sorting task, almost never justifies 10x the cost and slower runs. Ship the small model.

If the gap were 96% vs 78%, that's different. Then you'd either keep the big model or improve the small one's prompt.

A good free primer on evaluating models this way is OpenAI's Evals guide, which walks through building test sets and scoring outputs.

Choosing the Right LLM: Cost vs Quality

Watch on YouTube

Smart patterns that beat "just pick one"

You don't have to commit to a single model. Some of the best setups mix them.

Pattern 1: Cascade (cheap first, expensive on hard cases)

Run everything through the small model. When it's unsure, escalate to the big one.

You can ask the model for a confidence score, or check token probabilities where the API exposes them. If confidence is low, send that single item to the frontier model. If 90% of reviews are easy, you pay big-model prices on only 10% of the work.

Pattern 2: Batch mode for non-urgent jobs

If you don't need answers instantly, use batch processing. You submit a big pile of requests and get results back within a window (usually up to 24 hours) at roughly half price.

OpenAI, Anthropic, and Google all offer batch APIs. For our 2 million reviews, where nobody is waiting in real time, batch mode is perfect: latency doesn't matter, so you trade it for cost.

Pattern 3: Match latency to the human on the other end

  • A live chatbot where a person waits: latency matters a lot. Pick a fast model and keep reasoning effort low.
  • A nightly report that runs while everyone sleeps: latency barely matters. Optimize for cost or quality.

Ask: "Is a human staring at a loading spinner?" If no, you have room to save money.

Knowledge check

1. According to the lesson, what is the core principle behind matching a model to a task?

2. The lesson describes three things you're always trading when selecting a model. What is the key constraint it emphasizes about them?

3. Why do larger models tend to be both slower and more expensive per token?

MULTIPLE CHOICE

4. Select ALL reasons the classification (sentiment-sorting) task is a good fit for a small, fast model.

Select all the correct answers.

MULTIPLE CHOICE

5. Select ALL statements that correctly reflect the tradeoffs described in the lesson.

Select all the correct answers.

Common Mistakes

Defaulting to the biggest model "to be safe." This is the most expensive habit in AI projects. Safety here costs real money and adds delay for no quality gain on easy tasks.

Leaving reasoning on for trivial tasks. A reasoning model at default effort may think for hundreds of tokens before writing "positive." Turn effort down (reasoning_effort="minimal" or "low" on OpenAI, or leave extended thinking off on Claude) when the task doesn't need it.

Testing on three examples and calling it done. Three reviews tell you nothing. You need a test set big enough to trust, 100+ for a simple task.

Ignoring output length. Output tokens cost several times more than input tokens. A model that rambles is pricier than one that answers in one word. For classification, ask for one word and set a small output cap, leaving a little headroom for any reasoning tokens.

Forgetting that prompts can rescue small models. Often a small model fails because the prompt is vague, not because the model is weak. Adding one clear example (called a few-shot example) can close the gap:

Example:
Review: "Arrived broken and support ignored me."
Answer: negative

Now classify:
Review: "Works fine, nothing special."
Answer:

That one example can lift a small model's accuracy by several points, often enough to avoid paying for a bigger one.

A simple decision checklist

Before you pick a model, ask:

  1. How hard is the task really? Sorting and extracting are easy. Nuanced writing and multi-step reasoning are hard.
  2. How many requests? One-off means quality wins. Millions means cost and speed dominate.
  3. Is a human waiting? Yes means prioritize latency. No means batch it.
  4. Does it need reasoning? If not, turn it down or off.
  5. What's the accuracy floor? Decide the minimum acceptable score before testing, so you don't rationalize later.
  6. Did I actually measure? If you haven't run a test set, you're guessing.

Run those questions on the review job and the answer is obvious: easy task, huge volume, nobody waiting, small model clears the accuracy bar. Use the small one, in batch mode, with minimal reasoning and a tight output cap.

Key Takeaways

  • Match the model to the task, not your ego. For easy, high-volume work like classification, a small fast model usually beats a frontier model on cost and speed with almost no quality loss.
  • Hidden reasoning tokens are billed as output and add delay. Keep reasoning effort low on simple tasks.
  • Always test on a labeled set of 100+ examples before committing. Compare accuracy directly and set your acceptable floor in advance.
  • Use batch mode when no human is waiting to cut costs by roughly half, and cap output length (with max_completion_tokens on reasoning models) so you don't pay for rambling.
  • Reach for cascades (cheap model first, escalate hard cases) to get big-model quality on the few items that need it.
  • Improve the prompt before upgrading the model. A single few-shot example often closes the gap and saves you the upgrade entirely.

What to do, from this lesson

These actions are compiled in the role's Playbook.

  • Use small fast models in batch for high-volume easy work
See the full action playbook →

Related articles

Recent articles from the blog that build on this lesson.