Scoping, data, and success criteria

The tight version is buildable this week. It names the fields (the specific pieces of data you want), the input format (PDF), and the boundary (top 20 suppliers, English only). Everything outside that boundary is a "later" problem, and saying so out loud is part of the job.

Write down what's out of scope

Be explicit about exclusions. For the invoice tool:

• Handwritten invoices: out.
• Languages other than English: out for now.
• Multi-page invoices with line-item tables: out for v1.

Writing exclusions down prevents the slow creep where someone says "can it also just handle this one weird case?" and the project balloons.

Success criteria: define "good" as a number

Here is the rule: if you can't measure it, you can't finish it.

"Read invoices accurately" is not a success criterion. It is a wish. Turn it into something you can count.

> Success criterion: The tool extracts at least 95% of fields correctly across a test set of 100 real invoices.

Now "done" has a definition. You can run the tool, count the right answers, and get a number. Below 95%, keep working. At or above, ship.

How to pick the number

Don't invent 95% out of thin air. Anchor it to reality:

• What's the human baseline? If a tired employee typing at 5pm gets 97% of fields right, your bar is around there.
• What's the cost of an error? A wrong vendor name is annoying. A wrong total of $40,000 is a disaster. You may want a stricter target on high-stakes fields.

This leads to a smarter criterion that weights what matters:

> 99% correct on total and currency, 95% on the rest, and every low-confidence extraction flagged for a human to check.

That last part matters. A tool that says "I'm not sure about this one" is far safer than one that confidently guesses. Build the flag in from the start.

Decide how you'll measure before you build

You need a test set: a fixed collection of real examples with the correct answers filled in by a human. This is your answer key.

For the invoice tool, that means taking 100 real invoices and manually recording the five correct fields for each, in a spreadsheet. Yes, by hand. This is boring and it is the single highest-value hour of the project. Without it you are guessing.

The Google "People + AI Guidebook" has a clear, free section on defining success and reward functions for AI products if you want to go deeper.

Audit your data before you trust it

Now the second half: data. Your AI tool is only as good as what you feed it. Before building, audit what you actually have.

Open 30 real invoices and look. Don't theorize. Look. You will find surprises every time:

• 12 are clean digital PDFs.
• 9 are scans, slightly crooked, one is upside down.
• 5 are photos taken on a phone.
• 3 use a date format you didn't expect.
• 1 is a vendor's "invoice" that is actually a forwarded email.

This five-minute exercise just reshaped your project. Phone photos and scans need a different handling path than clean PDFs. The upside-down one will fail unless you plan for it.

Three questions for any dataset

1. Is it representative? Do your 30 samples match what the tool will see in production? If you only test on clean PDFs but 40% of real invoices are scans, your 95% score is a lie.

2. Is it labeled? Do you have the correct answers (the "labels") to check against? If not, that is your first task.

3. Is it allowed? Can you legally and ethically feed this data to an AI service? Invoices contain vendor names, bank details, sometimes personal info. Check whether you can send it to an external APIAPIApplication Programming Interface: a standardised interface that lets applications communicate and exchange data without knowing each other's internal workings.View full definition → or whether it must stay in-house.

That third point is real in 2026. Tools like ChatGPT, Claude, and Gemini offer business tiers that don't train on your data, but free consumer tiers may. Know which one you're using before uploading a customer's financial records.

Putting it together: a first runnable test

Once you have scope, a success number, and an audited test set, your first build is tiny: run a few invoices through a model and see how close you land. Modern models like Claude and Gemini accept a document and a prompt directly, so the "code" is mostly a clear instruction.

Here is a minimal example using the Anthropic APIAPIApplication Programming Interface: a standardised interface that lets applications communicate and exchange data without knowing each other's internal workings.View full definition →. It asks for the fields back as structured data (JSON), which makes scoring against your answer key easy.

import anthropic, json

client = anthropic.Anthropic()  # uses your API key from the environment

prompt = """Extract these fields from the invoice text below.
Return ONLY valid JSON with keys: vendor, invoice_number, date, total, currency.
If a field is missing or unclear, use null. Do not guess.

INVOICE TEXT:
{invoice_text}
"""

invoice_text = "ACME Corp  Invoice #4471  Date: 12 Jan 2026  Total: 1,250.00 USD"

msg = client.messages.create(
    model="claude-sonnet-4-5",
    max_tokens=300,
    messages=[{"role": "user", "content": prompt.format(invoice_text=invoice_text)}],
)

result = json.loads(msg.content[0].text)
print(result)
# {'vendor': 'ACME Corp', 'invoice_number': '4471',
#  'date': '2026-01-12', 'total': 1250.00, 'currency': 'USD'}

Two design choices here carry the lesson:

• "Do not guess." This pushes the model toward null instead of confident wrong answers, which feeds your human-review flag.
• Structured JSON output. Because the answer comes back as named fields, you can compare it to your spreadsheet answer key automatically and compute that 95% number.

Scoring against your answer key

The scoring loop is just counting matches:

def score(predicted, correct):
    fields = ["vendor", "invoice_number", "date", "total", "currency"]
    hits = sum(1 for f in fields if predicted.get(f) == correct[f])
    return hits / len(fields)

# Run this across all 100 test invoices, average the scores.
# That average IS your success metric.

Run it on all 100, average the results, and you have an honest number. Now you can make decisions: improve the prompt, handle scans separately, or ship.

🎬 [VIDEO: "How to Evaluate LLMLLMA Large Language Model is an AI system trained on vast text data to predict and generate language, enabling tasks like writing, summarizing, and answering questions.View full definition → Outputs" - youtube.com - a practical walkthrough of building a small test set and scoring model accuracy before you scale up]

The mindset

The whole point is to replace opinions with evidence. "It feels pretty good" becomes "it scores 91%, and the failures are all phone photos." That second sentence tells you exactly what to fix next.

Define good, audit your data, build the smallest thing that produces a number, then improve. Every serious AI project, from a one-person invoice tool to a company-wide system, runs on this loop.

Key Takeaways

• Start with the outcome, not the tool. Name the real-world result first ("stop typing invoices by hand"), then decide what AI must do to get there.
• Scope tight and write down exclusions. Five fields from English PDFs by your top 20 vendors ships this week. "All invoices" never ships.
• Make "good" a number with a test set. Hand-label 100 real examples, then measure against them. "95% of fields correct, total and currency at 99%" is a finish line; "accurate" is not.
• Audit data by looking at it. Open 30 real samples before building. Check that they're representative, labeled, and legally allowed in the AI tool you've chosen.
• Ask for structured output and "don't guess." JSON fields make scoring automatic, and flagging low-confidence cases for human review keeps costly errors out of production.