Evaluating and running proof-of-concepts with AI vendors

The risk is that extraction accuracy degrades sharply on the messy documents that make up your actual workload. A POC exists to expose that gap.

Phase 0: Before day one

Do not start the clock until three things are locked.

Define the exact scope. "Extract data from prospectuses" is too vague. Pick 8 to 15 specific fields: ongoing charges figure, benchmark, ISIN (International Securities Identification Number), inception date, minimum investment, redemption frequency, share class currency. Each field is a testable target.

Assemble a representative document set. Not the vendor's clean samples. Your documents. Include the ugly ones: scanned faxes, multi-language prospectuses, funds with 12 share classes in a single table. A common mistake is testing on 20 tidy PDFs when production will throw 20,000 varied ones.

Agree on the ground truth. You need a "gold set" of documents where a human has already recorded the correct answer for every field. Without this, you cannot measure accuracy. Budget real analyst hours to build it. This is the single most skipped step and the reason most POCs produce arguments instead of evidence.

Phase 1 (Days 1 to 30): Data access and integration

The first month is rarely about AI quality. It is about whether the vendor can even get to your data and return results in a usable form.

Key questions:

• Where does the data go? If prospectuses contain nonpublic information, does the vendor process them in your cloud tenant, or send them to a third-party APIAPIApplication Programming Interface: a standardised interface that lets applications communicate and exchange data without knowing each other's internal workings.View full definition →? This has direct implications under GDPR (the EU General Data Protection Regulation) and your own information security policy.
• How are results delivered? A JSON file, an APIAPIApplication Programming Interface: a standardised interface that lets applications communicate and exchange data without knowing each other's internal workings.View full definition → endpoint, a spreadsheet? It must plug into your downstream systems.
• What is the human-in-the-loop workflow? When the tool is unsure, how does it flag low-confidence extractions for review?

A good extraction result carries a confidence score per field, so your team reviews only the uncertain ones:

Here, the 0.62 confidence on charges is the tool telling you to check that field. A vendor whose tool returns no confidence scores is a red flag: it forces you to review everything or trust blindly.

Phase 2 (Days 31 to 60): Measuring accuracy that matters

Now run the full gold set and measure. The two metrics people confuse constantly:

• Precision: of the fields the tool filled in, what fraction were correct?
• Recall: of the fields that should have been filled, what fraction did the tool find?

For a fee field, high precision matters most: a wrong fee that looks confident is worse than a blank one, because a blank one gets reviewed.

A worked example

Suppose your gold set has 1,000 prospectuses and you test the "ongoing charges figure" field.

• The tool extracted a value for 950 documents.
• Of those 950, it was correct on 900.
• The other 50 documents it left blank (flagged for human review).

Precision = 900 / 950 = 94.7%

Recall (against all 1,000) = 900 / 1,000 = 90%

Now the business question: is 94.7% precision good enough? For a field that feeds a regulatory disclosure, a 5.3% error rate on 950 documents is roughly 50 wrong fees going out. Probably unacceptable without a review layer. For an internal research database, it might be fine.

This is the core insight: there is no universal accuracy bar. The bar depends on what the extracted data does next.

Compare against the honest baseline

The right comparison is not "AI versus perfection." It is "AI plus human review versus your current process." Your manual process is not 100% accurate either. If analysts currently make errors on 3% of manually keyed fields, a tool at 94.7% precision with review may already beat the status quo on both cost and quality.

For a structured framework on evaluating machine learning claims, the Google People + AI Guidebook is a free, non-technical resource on setting the right expectations with AI systems.

Phase 3 (Days 61 to 90): Cost, scale, and the exit decision

You now know the tool works on your data. The final month answers whether it works at your volume and price.

Throughput. If the POC processed 1,000 documents comfortably, what happens at 50,000 per quarter? Ask the vendor for a load test.

Cost model. Many AI vendors price per document, per page, or per tokentokenA token is the basic unit of text that language models process, often a word fragment, whole word, or punctuation mark rather than a single character.View full definition → (a tokentokenA token is the basic unit of text that language models process, often a word fragment, whole word, or punctuation mark rather than a single character.View full definition → is roughly three-quarters of a word for an LLMLLMA Large Language Model is an AI system trained on vast text data to predict and generate language, enabling tasks like writing, summarizing, and answering questions.View full definition →). A prospectus can run 100 pages. Run the actual arithmetic:

If the vendor charges 0.02 US dollars per page and your prospectuses average 120 pages, that is 2.40 dollars per document. At 50,000 documents per year, that is 120,000 dollars annually in processing alone, before license fees and your review labor. (Figures illustrative, not a quoted price.)

Now compare to the labor saved. If a manual analyst takes 25 minutes per prospectus and the tool plus review takes 5 minutes, you save 20 minutes on 50,000 documents, roughly 16,600 hours. That is the ROIROIReturn on Investment: the ratio of net profit to the cost of an investment. A 300% ROI means each dollar invested returns $3.View full definition → conversation, and it must include your review time, not just the license.

Writing exit criteria before you sign

The whole point of a 90-day POC is to be able to walk away. Write the exit criteria into the POC agreement itself, in advance, so the decision is not emotional or political after everyone has invested effort.

Good exit criteria are numeric and pre-committed:

• Precision on the fee field must reachreachThe number of unique people exposed to your message in a given period. Unlike impressions, reach counts each person once, no matter how often they see it.View full definition → at least 93% on the gold set.
• Recall across all target fields must reachreachThe number of unique people exposed to your message in a given period. Unlike impressions, reach counts each person once, no matter how often they see it.View full definition → at least 85%.
• Confidence scores must be well-calibrated (when the tool says 0.9, it should be right about 90% of the time).
• End-to-end processing including review must cost less per document than the current manual process.
• Integration with the target system must be demonstrated, not promised.

If the tool misses these, you do not sign. Notice that "the demo was impressive" is not on the list.

Common vendor red flags during a POC

• They resist testing on your messy documents and push their curated samples.
• They cannot explain how the model handles a document type it has not seen.
• They report accuracy without telling you the test set or the metric used.
• Pricing is quoted only after you are committed.
• No answer on where your data physically goes.

A confident vendor welcomes a hard POC. It is how good tools win.

Key Takeaways

• A demo is marketing; a POC is evidence. Never sign on a demo. Insist on a structured, time-boxed test on your own documents.
• Build the gold set first. Without human-verified ground truth on your real documents, you cannot measure accuracy, and every disagreement becomes opinion.
• Precision and recall are not the same, and the required bar depends on what the data feeds. A regulatory disclosure needs a higher bar than an internal database.
• Compare AI plus human review against your actual current process, not against perfection. Include review labor and full processing cost in the ROIROIReturn on Investment: the ratio of net profit to the cost of an investment. A 300% ROI means each dollar invested returns $3.View full definition →, not just the license fee.
• Write numeric exit criteria before the POC starts. Pre-committed thresholds let you walk away without politics when the tool underperforms.