Cost, latency, and model selection tradeoffs

A startup ran 2 million customer reviews through GPT-5 to sort them into "positive," "negative," or "neutral." It worked great. It also took 9 hours and cost about $4,000. They switched to a small, fast model. Same job: 40 minutes, about $80, and accuracy dropped by less than 1%.

That is the whole lesson in one story. The biggest model is rarely the right one for high-volume, repetitive work. Let me show you how to think about it.

Three things you're always trading

Every time you pick a model, you balance three things:

• Quality: how good the output is.
• Latency: how long you wait for a response. (Latency just means delay.)
• Cost: how much you pay per request.

You can usually optimize for two, but not all three. A huge frontier model gives you top quality but costs more and runs slower. A small model is cheap and fast but may stumble on hard tasks.

The skill is matching the model to the job, not always reaching for the most powerful one.

Why bigger models are slower and pricier

Large language models charge by the tokentokenA token is the basic unit of text that language models process, often a word fragment, whole word, or punctuation mark rather than a single character.View full definition →, a token being roughly 3/4 of a word. "Customer service" is about 4 tokens.

Bigger models have more internal "parameters" (the dials they tune during training). More parameters means more computation per token, which means more cost and more delay. You are literally paying for more math on every word.

So a 10x bigger model can cost 10x to 30x more per token and respond several times slower. That difference is invisible on one request. On 2 million requests, it is your whole budget.

A concrete example: the classification job

Let's make the review-sorting task real.

You have 2 million reviews. Each is short (say 50 tokens in, 1 token out, just the label). You want each tagged positive, negative, or neutral. This is classification: sorting items into fixed buckets.

Here is the prompt you'd send for each review:

Classify the sentiment of this review as exactly one word:
positive, negative, or neutral.

Review: "Shipping was slow but the product is excellent."

Answer:

This is an easy task. Sentiment is something even small models handle well. You do not need a model that can write poetry or debug code. You need one that reads three sentences and outputs one word, fast and cheap.

Run the numbers

Rough 2026 pricing (always check current rates, they move):

For 2 million reviews at ~50 tokens each, that's 100 million input tokens.

• Frontier model: roughly $300 to $1,500, and many hours.
• Small model: roughly $10 to $40, and a fraction of the time.

The small model wins decisively, as long as accuracy holds. And for a task this simple, it usually does. That is the catch you must verify, which brings us to testing.

How to actually decide: test, don't guess

Never assume the small model is good enough. Measure it. Here's the workflow.

Step 1: Build a small labeled test set

Hand-label 100 to 200 reviews yourself. This is your ground truth: the correct answers you compare against.

Step 2: Run both models on the test set

from openai import OpenAI
client = OpenAI()

def classify(review, model):
    resp = client.chat.completions.create(
        model=model,
        messages=[{
            "role": "user",
            "content": f"Classify sentiment as one word "
                       f"(positive, negative, neutral): {review}"
        }],
        max_tokens=1,
    )
    return resp.choices[0].message.content.strip().lower()

# Compare a small model vs a large one
for model in ["gpt-5-mini", "gpt-5"]:
    correct = sum(
        classify(r["text"], model) == r["label"]
        for r in test_set
    )
    print(model, correct / len(test_set))

Step 3: Compare accuracy and decide

Say the big model scores 96% and the small one scores 95%. That 1% gap, on a review-sorting task, almost never justifies 10x the cost and slower runs. Ship the small model.

If the gap were 96% vs 78%, that's different. Then you'd either keep the big model or improve the small one's prompt.

A great free primer on evaluating models this way is OpenAI's Evals guide, which walks through building test sets and scoring outputs.

Smart patterns that beat "just pick one"

You don't have to commit to a single model. Some of the best setups mix them.

Pattern 1: Cascade (cheap first, expensive on hard cases)

Run everything through the small model. When it's unsure, escalate to the big one.

Many models can return a confidence signal or you can ask for one. If confidence is low, send that single item to the frontier model. If 90% of reviews are easy, you pay big-model prices on only 10% of the work.

Pattern 2: Batch mode for non-urgent jobs

If you don't need answers instantly, use batch processing. You submit a big pile of requests and get results back within a window (often 24 hours) at roughly half price.

OpenAI, Anthropic, and Google all offer batch APIs. For our 2 million reviews, where nobody is waiting in real time, batch mode is perfect: latency doesn't matter, so you trade it for cost.

Pattern 3: Match latency to the human on the other end

• A live chatbot where a person waits: latency matters a lot. Pick a fast model.
• A nightly report that runs while everyone sleeps: latency barely matters. Optimize for cost or quality.

Ask: "Is a human staring at a loading spinner?" If no, you have room to save money.

Common Mistakes

Defaulting to the biggest model "to be safe." This is the most expensive habit in AI projects. Safety here costs real money and adds delay for no quality gain on easy tasks.

Testing on three examples and calling it done. Three reviews tell you nothing. You need a test set big enough to trust, 100+ for a simple task.

Ignoring output length. Output tokens often cost more than input tokens. A model that rambles is pricier than one that answers in one word. For classification, cap it: max_tokens=1 forces a short answer and saves money.

Forgetting that prompts can rescue small models. Often a small model fails not because it's weak but because the prompt is vague. Adding one clear example (called a few-shot example) can close the gap:

Example:
Review: "Arrived broken and support ignored me."
Answer: negative

Now classify:
Review: "Works fine, nothing special."
Answer:

That one example can lift a small model's accuracy by several points, often enough to avoid paying for a bigger one.

A simple decision checklist

Before you pick a model, ask:

1. How hard is the task really? Sorting and extracting are easy. Nuanced writing and multi-step reasoning are hard.
2. How many requests? One-off means quality wins. Millions means cost and speed dominate.
3. Is a human waiting? Yes means prioritize latency. No means batch it.
4. What's the accuracy floor? Decide the minimum acceptable score before testing, so you don't rationalize later.
5. Did I actually measure? If you haven't run a test set, you're guessing.

Run those questions on the review job and the answer is obvious: easy task, huge volume, nobody waiting, small model clears the accuracy bar. Use the small one, in batch mode, with max_tokens=1.

Key Takeaways

• Match the model to the task, not your ego. For easy, high-volume work like classification, a small fast model usually beats a frontier model on cost and speed with almost no quality loss.
• Always test on a labeled set of 100+ examples before committing. Compare accuracy directly and set your acceptable floor in advance.
• Use batch mode when no human is waiting to cut costs by roughly half, and cap output length to avoid paying for rambling.
• Reach for cascades (cheap model first, escalate hard cases) to get big-model quality on the few items that need it without paying for all of them.
• Improve the prompt before upgrading the model. A single few-shot example often closes the gap and saves you the upgrade entirely.