Cost, latency, and model selection tradeoffs
A startup ran 2 million customer reviews through a frontier reasoning model (GPT-5 at default settings) to sort them into "positive," "negative," or "neutral." It worked great. It also took 9 hours and cost about $1,500, most of it spent on hidden "thinking" tokenstokensA token is the basic unit of text that language models process, often a word fragment, whole word, or punctuation mark rather than a single character.View full definition → nobody needed. They switched to a small, fast model with reasoning turned down. Same job: 40 minutes, about $40, and accuracy dropped by less than 1%.
That is the whole lesson in one story. The biggest model is rarely the right one for high-volume, repetitive work. Here is how to think about it.
Three things you're always trading
Every time you pick a model, you balance three things:
- Quality: how good the output is.
- Latency: how long you wait for a response. (Latency just means delay.)
- Cost: how much you pay per request.
You can usually optimize for two, but not all three. A huge frontier model gives you top quality but costs more and runs slower. A small model is cheap and fast but may stumble on hard tasks.
The skill is matching the model to the job, not always reaching for the most powerful one.
Why bigger models are slower and pricier
Large language models charge by the token, a token being roughly 3/4 of a word. "Customer service" is about 2 to 3 tokens.
Bigger models have more internal "parameters" (the dials they tune during training). More parameters means more computation per token, which means more cost and more delay.
There is a second cost that is easy to miss. Many current models (GPT-5, Claude with extended thinking, Gemini 2.5 Pro) are reasoning models: before answering, they generate hidden reasoning tokens. You never see them, but you pay for them at the output rate and you wait for them. On a one-word answer, a model might "think" for hundreds of tokens first.
So a frontier model can easily cost 10x to 50x more per request than a small one and respond several times slower. That difference is invisible on one request. On 2 million requests, it is your whole budget.
A concrete example: the classification job
Let's make the review-sorting task real.
You have 2 million reviews. Each is short (say 50 tokens in, one word out, just the label). You want each tagged positive, negative, or neutral. This is classification: sorting items into fixed buckets.
Here is the prompt you'd send for each review:
Classify the sentiment of this review as exactly one word:
positive, negative, or neutral.
Review: "Shipping was slow but the product is excellent."
Answer:This is an easy task. Even small models handle sentiment well. You do not need a model that can write poetry or debug code. You need one that reads three sentences and outputs one word, fast and cheap.
Run the numbers
Rough late-2026 pricing (always check current rates, they move often):
| Model type | Cost per 1M input tokens | Cost per 1M output tokens | Relative speed |
|---|---|---|---|
| Frontier (GPT-5, Claude Opus, Gemini Pro) | ~$1.25 to $5 | ~$10 to $25 | Slower |
| Small/fast (GPT-5 mini or nano, Claude Haiku, Gemini Flash) | ~$0.05 to $1 | ~$0.40 to $5 | Much faster |
For 2 million reviews at ~50 tokens each, that's 100 million input tokens.
- Frontier model: roughly $125 to $500 for input alone. Add a few hundred hidden reasoning tokens per review at output prices and the bill climbs into the thousands, plus many hours.
- Small model with reasoning kept minimal: roughly $5 to $100, in a fraction of the time.
The small model wins decisively, as long as accuracy holds. For a task this simple, it usually does. That is the catch you must verify, which brings us to testing.
How to actually decide: test, don't guess
Never assume the small model is good enough. Measure it. Here's the workflow.
Step 1: Build a small labeled test set
Hand-label 100 to 200 reviews yourself. This is your ground truth: the correct answers you compare against.
Step 2: Run both models on the test set
from openai import OpenAI
client = OpenAI()
def classify(review, model):
resp = client.chat.completions.create(
model=model,
messages=[{
"role": "user",
"content": f"Classify sentiment as one word "
f"(positive, negative, neutral): {review}"
}],
reasoning_effort="minimal", # skip most hidden thinking
max_completion_tokens=16, # includes reasoning tokens
)
return (resp.choices[0].message.content or "").strip().lower()
# Compare a small model vs a large one
for model in ["gpt-5-mini", "gpt-5"]:
correct = sum(
classify(r["text"], model) == r["label"]
for r in test_set
)
print(model, correct / len(test_set))Two details matter here. Reasoning models reject the old max_tokens parameter, so you use max_completion_tokens. And that cap counts hidden reasoning tokens too. Set it to 1 and the model may spend its single token thinking and return nothing at all.
Step 3: Compare accuracy and decide
Say the big model scores 96% and the small one scores 95%. That 1% gap, on a review-sorting task, almost never justifies 10x the cost and slower runs. Ship the small model.
If the gap were 96% vs 78%, that's different. Then you'd either keep the big model or improve the small one's prompt.
A good free primer on evaluating models this way is OpenAI's Evals guide, which walks through building test sets and scoring outputs.
Choosing the Right LLM: Cost vs Quality
Smart patterns that beat "just pick one"
You don't have to commit to a single model. Some of the best setups mix them.
Pattern 1: Cascade (cheap first, expensive on hard cases)
Run everything through the small model. When it's unsure, escalate to the big one.
You can ask the model for a confidence score, or check token probabilities where the APIAPIApplication Programming Interface: a standardised interface that lets applications communicate and exchange data without knowing each other's internal workings.View full definition → exposes them. If confidence is low, send that single item to the frontier model. If 90% of reviews are easy, you pay big-model prices on only 10% of the work.
Pattern 2: Batch mode for non-urgent jobs
If you don't need answers instantly, use batch processing. You submit a big pile of requests and get results back within a window (usually up to 24 hours) at roughly half price.
OpenAI, Anthropic, and Google all offer batch APIs. For our 2 million reviews, where nobody is waiting in real time, batch mode is perfect: latency doesn't matter, so you trade it for cost.
Pattern 3: Match latency to the human on the other end
- A live chatbot where a person waits: latency matters a lot. Pick a fast model and keep reasoning effort low.
- A nightly report that runs while everyone sleeps: latency barely matters. Optimize for cost or quality.
Ask: "Is a human staring at a loading spinner?" If no, you have room to save money.
Knowledge check
1. According to the lesson, what is the core principle behind matching a model to a task?
2. The lesson describes three things you're always trading when selecting a model. What is the key constraint it emphasizes about them?
3. Why do larger models tend to be both slower and more expensive per token?
4. Select ALL reasons the classification (sentiment-sorting) task is a good fit for a small, fast model.
Select all the correct answers.
5. Select ALL statements that correctly reflect the tradeoffs described in the lesson.
Select all the correct answers.
Common Mistakes
Defaulting to the biggest model "to be safe." This is the most expensive habit in AI projects. Safety here costs real money and adds delay for no quality gain on easy tasks.
Leaving reasoning on for trivial tasks. A reasoning model at default effort may think for hundreds of tokens before writing "positive." Turn effort down (reasoning_effort="minimal" or "low" on OpenAI, or leave extended thinking off on Claude) when the task doesn't need it.
Testing on three examples and calling it done. Three reviews tell you nothing. You need a test set big enough to trust, 100+ for a simple task.
Ignoring output length. Output tokens cost several times more than input tokens. A model that rambles is pricier than one that answers in one word. For classification, ask for one word and set a small output cap, leaving a little headroom for any reasoning tokens.
Forgetting that prompts can rescue small models. Often a small model fails because the prompt is vague, not because the model is weak. Adding one clear example (called a few-shot example) can close the gap:
Example:
Review: "Arrived broken and support ignored me."
Answer: negative
Now classify:
Review: "Works fine, nothing special."
Answer:That one example can lift a small model's accuracy by several points, often enough to avoid paying for a bigger one.
A simple decision checklist
Before you pick a model, ask:
- How hard is the task really? Sorting and extracting are easy. Nuanced writing and multi-step reasoning are hard.
- How many requests? One-off means quality wins. Millions means cost and speed dominate.
- Is a human waiting? Yes means prioritize latency. No means batch it.
- Does it need reasoning? If not, turn it down or off.
- What's the accuracy floor? Decide the minimum acceptable score before testing, so you don't rationalize later.
- Did I actually measure? If you haven't run a test set, you're guessing.
Run those questions on the review job and the answer is obvious: easy task, huge volume, nobody waiting, small model clears the accuracy bar. Use the small one, in batch mode, with minimal reasoning and a tight output cap.
Key Takeaways
- Match the model to the task, not your ego. For easy, high-volume work like classification, a small fast model usually beats a frontier model on cost and speed with almost no quality loss.
- Hidden reasoning tokens are billed as output and add delay. Keep reasoning effort low on simple tasks.
- Always test on a labeled set of 100+ examples before committing. Compare accuracy directly and set your acceptable floor in advance.
- Use batch mode when no human is waiting to cut costs by roughly half, and cap output length (with
max_completion_tokenson reasoning models) so you don't pay for rambling. - ReachReachThe number of unique people exposed to your message in a given period. Unlike impressions, reach counts each person once, no matter how often they see it.View full definition → for cascades (cheap model first, escalate hard cases) to get big-model quality on the few items that need it.
- Improve the prompt before upgrading the model. A single few-shot example often closes the gap and saves you the upgrade entirely.
What to do, from this lesson
These actions are compiled in the role's Playbook.
- Use small fast models in batch for high-volume easy work
Related articles
Recent articles from the blog that build on this lesson.
- AIGoogle shipped a Gemini that never writes a sentence, and it changes how you pick a modelGoogle's Gemini Decision 1 only scores answers you write in advance, and OpenAI already sells the same kind of model, so asking "ChatGPT, Claude or Gemini?" no longer gets you far. This playbook shows how to pick a model for each job, test it on your own cases and price the whole task.
- AIAI spend per employee is falling: efficiency win or adoption stall?Enterprise AI spending per employee dropped at top firms in August 2026, prompting talk of a slowdown. The real story is more complicated, and more interesting, than either the optimists or the pessimists want to admit.
- AIWho's actually cracking AI ROI: a field guide to the standoutsMost AI ROI conversations produce more heat than light, mixing vendor case studies with genuine breakthroughs and calling it insight. This field guide cuts through to the companies, researchers, and milestones actually worth studying if you want to understand what separates real returns from expensive experiments.