Evaluating AI apps before you trust them: a practical playbook

0:00Welcome to the Leaders Insights podcast.

0:07Today's episode, evaluating AI apps before you trust them.

0:11A practical playbook.

0:13Every vendor demo I've ever seen works perfectly.

0:16Then the thing hits production and falls apart.

0:19What's the trick they're pulling?

0:20The trick is, they built the demo on the 10 questions they know it answers well.

0:25It's like test driving a car on a track the dealer paved themselves.

0:29The moment you hand it your real data, your real edge cases, the polish comes off.

0:34So how does a smart exec who doesn't code test this before signing a contract?

0:38You start with a golden set.

0:40That's just 50 to 100 real questions from your business, with answers you already know

0:44are correct.

0:46Not vendor questions, oh, yours.

0:48You run the tool against them and count how often it's right.

0:5150 questions sounds trivial.

0:54Why not thousands?

0:55Because 50 real ones beat a thousand you'll never read.

0:58And here's the discipline.

1:00Pick the questions that hurt.

1:02The ambiguous ones, the ones where two policies contradict each other.

1:06The ones a new hire would get wrong.

1:08If it only nails the easy stuff, you've bought an expensive parrot.

1:12Most of these enterprise tools are rag-based, right?

1:16Explain that before we go further.

1:17Rag.

1:18Retrieval augmented generation.

1:20Where the AI looks up your documents first and then writes an answer from what it found.

1:26The idea is, it quotes your handbook instead of making things up.

1:31In theory, it's grounded.

1:33In practice, it depends entirely on whether the retrieval step actually fetched the right

1:38document.

1:39And if it fetches the wrong one?

1:40Then it writes a confident, beautifully worded answer based on garbage.

1:44That's the failure mode nobody demos.

1:47So the second test is, for each answer, does it show you the source?

1:51A good tool cites the exact passage.

1:54If it can't tell you where the answer came from, you're trusting vibes.

1:58Okay, retrieval and sourcing.

2:00What's the third thing people skip?

2:02Hallucination under pressure.

2:04You deliberately ask it things your documents don't cover.

2:07The right answer is, I don't know.

2:09The wrong answer is a fluent paragraph of invention.

2:12I've watched a legal AI tool from a well-known vendor cite a statute that doesn't exist

2:18with a fake case number.

2:20Their own benchmark claimed 95% accuracy, which tells you their test set was softer

2:25than a hotel pillow.

2:26That's a vendor number, though.

2:28How much should I trust any accuracy figure they hand me?

2:31Treat every vendor accuracy claim as marketing until you reproduce it.

2:35Ask one question.

2:37What was in your test set?

2:39If they won't share it, the number is theater.

2:41The 2026 releases love throwing around enterprise grade.

2:45That phrase means nothing you can measure.

2:47Say the tool passes accuracy and sourcing.

2:50Am I done?

2:51No.

2:52Now you test the boring stuff that kills you later.

2:55Latency.

2:56How long an answer takes.

2:58Cost per query at your real volume, not the pilots.

3:01And consistency.

3:03Ask the same question five times.

3:05See if you get five different answers.

3:07Generative AI is non-deterministic, meaning it can reword or even change its answer run

3:12to run.

3:14For a marketing blurb, fine.

3:16For a compliance policy, that's a lawsuit waiting.

3:19How do you test costs realistically before you've deployed it?

3:22You take your golden set, multiply by your expected daily volume, and make the vendor

3:27quote you against that.

3:29A tool that's cheap at a thousand queries a day can be brutal at a hundred thousand.

3:35Companies discover this when the first month's bill lands and someone's face goes white.

3:39What about the tool getting worse over time?

3:42Does that happen?

3:43Constantly, the vendor updates the underlying model and behavior shifts overnight.

3:48So keep your golden set and rerun it monthly after you deploy.

3:52It's your smoke detector.

3:54The teams that skip this find out from angry customers instead.

3:58Always the more expensive way to learn.

4:00If someone does one thing after this episode, what is it?

4:03Build the golden set this week.

4:0450 real questions, known answers, the nasty ones included.

4:09Before you watch a single demo.

4:11Because if you walk in with your own test, the vendor can't steer you to their paved

4:14track.

4:15You control the ground.

4:17And suddenly, the honest tools look very different from the loud ones.

4:21That's it.

4:22The AI Decision Tools are live at MBA-training.com.