Evaluating outputs: how do you know it works?

# Evaluating outputs: how do you know it works?

You built a tool that summarizes customer reviews. You tested it twice, it looked great, you shipped it. Three weeks later someone shows you a summary that says a five-star review was "negative." How long had that been happening? You have no idea, because you never measured it.

This is the trap. "It looked fine" is a feeling, not a metric. To improve an AI system, you need a way to score it that does not depend on your mood that afternoon.

That tool is called an eval (short for "evaluation"): a repeatable test that measures how well your AI does a specific job.

Why "looks fine" fails

Large language models are non-deterministic. That means the same prompt can give slightly different answers each time. So checking one output tells you almost nothing about the next hundred.

You also change things constantly: you tweak a prompt, switch from GPT-4o to a cheaper model, add an instruction. Every change can help one case and quietly break another. Without a score, you are flying blind.

An eval fixes this. You build a small set of test cases once, then run them every time you change something. The score goes up or down. Now you can actually improve.

The core idea: a test set

A test set is a collection of example inputs paired with what a good output looks like. For a summarizer, the simplest version is a set of question-and-answer pairs.

Here is the move. Instead of trying to grade a whole summary (hard and fuzzy), you write specific questions that a correct summary must be able to answer.

Say you are summarizing this product review:

> "The blender is powerful and crushes ice easily, but it is extremely loud and the lid leaks if you fill it past the halfway line. Shipping took two weeks."

You write questions whose answers should survive summarization:

| Question | Expected answer |

|---|---|

| Is the blender powerful? | Yes |

| What is the main complaint about noise? | It is loud |

| Does the lid have an issue? | Yes, it leaks when overfilled |

| Was shipping fast? | No, took two weeks |

If the summary lets you answer all four correctly, it kept the important facts. If it drops the leaking lid, you catch it.

This turns a vague question ("is this a good summary?") into a concrete, countable one ("how many of these facts did it preserve?").

Building a real eval, step by step

Let's build a tiny one. You need three things:

1. Inputs: a handful of reviews (start with 10 to 20, not 1,000).

2. Checks: question-and-answer pairs for each input.

3. A scorer: something that decides pass or fail.

Step 1: collect inputs and expected facts

Store them as plain data. No code background needed to read this:

test_cases = [
    {
        "review": "The blender is powerful and crushes ice easily, "
                  "but it is extremely loud and the lid leaks if you "
                  "fill it past the halfway line. Shipping took two weeks.",
        "must_include": ["loud", "leak", "powerful"],
    },
    {
        "review": "Battery lasts all day and the screen is bright. "
                  "Setup was confusing and customer support never replied.",
        "must_include": ["battery", "setup", "support"],
    },
]

The must_include list is a simple version of your "answers": key facts the summary must mention.

Step 2: generate the summaries

Here we call a model. This example uses the OpenAI APIAPIApplication Programming Interface: a standardised interface that lets applications communicate and exchange data without knowing each other's internal workings.View full definition →, but the pattern is identical for Claude or Gemini.

from openai import OpenAI
client = OpenAI()

def summarize(review):
    response = client.chat.completions.create(
        model="gpt-4o-mini",
        messages=[
            {"role": "system", "content": "Summarize this product review in one sentence. Keep the key pros and cons."},
            {"role": "user", "content": review},
        ],
    )
    return response.choices[0].message.content

Step 3: score it

The simplest possible scorer: did the summary contain each required keyword?

def score(summary, must_include):
    hits = [word for word in must_include if word.lower() in summary.lower()]
    return len(hits) / len(must_include)

total = 0
for case in test_cases:
    summary = summarize(case["review"])
    s = score(summary, case["must_include"])
    total += s
    print(f"Score: {s:.2f} | {summary}")

print(f"\nAverage score: {total / len(test_cases):.2f}")

Run it. You might get an average of 0.83. That number is your baseline. Now change your prompt, run again, and watch the number move. Up is good. Down means you just broke something, and you caught it before your users did.

Keyword matching is crude. That's okay to start.

Checking for the word "leak" misses a summary that says "the lid lets liquid through." Keyword scoring is blunt, but it is free, fast, and catches obvious regressions. Start here.

When you outgrow it, the next step is an LLM-as-judge: you ask a second AI model to grade the output. You give it the review, the summary, and a rubric, and ask for a score.

def judge(review, summary):
    prompt = f"""Original review: {review}

Summary: {summary}

Does the summary capture the main pros AND cons? 
Answer only PASS or FAIL."""
    response = client.chat.completions.create(
        model="gpt-4o",
        messages=[{"role": "user", "content": prompt}],
    )
    return response.choices[0].message.content.strip()

This catches meaning, not just exact words. The catch: the judge is also an AI, so it can be wrong. Spot-check its grades against your own judgment on a few cases before you trust it.

Reading your scores like a grown-up

A single number is a start, but how you look at it matters.

Track it over time. Save each run's average. A spreadsheet works fine. When the score drops from 0.85 to 0.72 after you "improved" the prompt, you revert.

Look at the failures, not the average. The 0.83 average hides which cases failed. Always read the worst ones. That is where you learn what your AI struggles with: long reviews, sarcasm, multiple languages.

Add hard cases on purpose. When a user reports a bad summary, do not just fix it. Add it to your test set as a permanent case. Your eval gets smarter every time something goes wrong. This collection is sometimes called a golden set: your trusted bank of examples.

Decide your passing bar. Is 0.83 good enough to ship? That depends on the stakes. Summarizing memes: sure. Summarizing medical instructions: absolutely not. You set the threshold based on the cost of being wrong.

A quick word on mindset

Evals feel like extra work when you are excited to ship. They are not overhead. They are the thing that lets you move fast without breaking things silently.

Think of it like cooking. Tasting the dish once is "it looked fine." A recipe with measured ingredients and a timer is an eval. The recipe is what lets you make the same good meal a hundred times, and improve it deliberately.

Start absurdly small. Ten test cases and a keyword check beat zero. You can always grow it.

If you want a deeper, still-readable guide, the OpenAI Evals documentation walks through running structured evals with their tooling, and the concepts transfer to any model you use.