Scoping, data, and success criteria

# Scoping, data, and success criteria

A team once spent six weeks building an AI tool to read invoices. It worked beautifully in demos. Then it hit real invoices: scanned receipts, currencies in three formats, a vendor who wrote dates as "3.4.26." Nobody had agreed on what "working" meant, so nobody could say whether the tool was done. It shipped late, then got pulled.

This lesson is about avoiding that. Before you write a line of code or a single prompt, you decide two things: what "good" looks like (your success criteria) and what data you actually have. We will use one running example, an invoice-extraction tool, the whole way through.

The mistake almost everyone makes

Most people start an AI project with the solution: "Let's use GPT to read our invoices." That is the wrong first move.

Start with the outcome instead. An outcome is the result you want in the real world, stated plainly:

> "We want to stop manually typing invoice totals into our accounting system. A person spends 4 hours a day on it."

Notice this says nothing about AI. That is good. Now you can ask: what would AI need to do to fix this, and how would we know it worked?

Scoping: shrink the problem until it's concrete

Scoping means drawing a tight boundary around exactly what your tool will and will not do. Broad scope kills projects. Narrow scope ships them.

Vague scope: "Process all our invoices automatically."

Tight scope: "Pull five fields (vendor name, invoice number, date, total, currency) from PDF invoices sent by our top 20 suppliers, in English, into a spreadsheet."

Putting it together: a first runnable test

Once you have scope, a success number, and an audited test set, your first build is tiny: run a few invoices through a model and see how close you land. Modern models like Claude and Gemini accept a document and a prompt directly, so the "code" is mostly a clear instruction.

Here is a minimal example using the Anthropic API. It asks for the fields back as structured data (JSON), which makes scoring against your answer key easy.

python

import anthropic, json

client = anthropic.Anthropic()  # uses your API key from the environment

prompt = """Extract these fields from the invoice text below.
Return ONLY valid JSON with keys: vendor, invoice_number, date, total, currency.
If a field is missing or unclear, use null. Do not guess.

INVOICE TEXT:
{invoice_text}
"""

invoice_text = "ACME Corp  Invoice #4471  Date: 12 Jan 2026  Total: 1,250.00 USD"

msg = client.messages.create(
    model="claude-sonnet-4-5",
    max_tokens=300,
    messages=[{"role": "user", "content": prompt.format(invoice_text=invoice_text)}],
)

result = json.loads(msg.content[0].text)
print(result)
# {'vendor': 'ACME Corp', 'invoice_number': '4471',
#  'date': '2026-01-12', 'total': 1250.00, 'currency': 'USD'}

Two design choices here carry the lesson:

"Do not guess." This pushes the model toward null instead of confident wrong answers, which feeds your human-review flag.

Structured JSON output. Because the answer comes back as named fields, you can compare it to your spreadsheet answer key automatically and compute that 95% number.

Scoring against your answer key

The scoring loop is just counting matches:

python

def score(predicted, correct):
    fields = ["vendor", "invoice_number", "date", "total", "currency"]
    hits = sum(1 for f in fields if predicted.get(f) == correct[f])
    return hits / len(fields)

# Run this across all 100 test invoices, average the scores.
# That average IS your success metric.

Run it on all 100, average the results, and you have an honest number. Now you can make decisions: improve the prompt, handle scans separately, or ship.

🎬 [VIDEO: "How to Evaluate LLM Outputs" - youtube.com - a practical walkthrough of building a small test set and scoring model accuracy before you scale up]

The mindset

The whole point is to replace opinions with evidence. "It feels pretty good" becomes "it scores 91%, and the failures are all phone photos." That second sentence tells you exactly what to fix next.

Define good, audit your data, build the smallest thing that produces a number, then improve. Every serious AI project, from a one-person invoice tool to a company-wide system, runs on this loop.

Key Takeaways

Start with the outcome, not the tool. Name the real-world result first ("stop typing invoices by hand"), then decide what AI must do to get there.

Scope tight and write down exclusions. Five fields from English PDFs by your top 20 vendors ships this week. "All invoices" never ships.

Make "good" a number with a test set. Hand-label 100 real examples, then measure against them. "95% of fields correct, total and currency at 99%" is a finish line; "accurate" is not.

Audit data by looking at it. Open 30 real samples before building. Check that they're representative, labeled, and legally allowed in the AI tool you've chosen.

Ask for structured output and "don't guess." JSON fields make scoring automatic, and flagging low-confidence cases for human review keeps costly errors out of production.

Scoping, data, and success criteria

The mistake almost everyone makes

Scoping: shrink the problem until it's concrete

Write down what's out of scope

Success criteria: define "good" as a number

How to pick the number

Decide how you'll measure before you build

Audit your data before you trust it

Three questions for any dataset

Putting it together: a first runnable test

Scoring against your answer key

The mindset

Key Takeaways

What to do, from this lesson