Grounding AI in your company's knowledge: a practical playbook

August 14, 2026 · 5 min · Free, with full transcript

Generic AI gives generic answers. This playbook shows you how to connect large language models to your organisation's own data so that every response is accurate, specific, and actually useful.

Listen to the podcast

4 min

Read the full transcript

Welcome to the Leaders Insights Podcast.

Today's episode, Grounding AI in Your Company's Knowledge, a practical playbook.

Every company I talked to bought an AI tool last year and got the same disappointment.

It sounds smart and knows nothing about their business.

Why?

Because they bought a brilliant intern who read the entire internet and never read a single

one of your documents.

A general model is trained on public text up to some cutoff date.

It has no idea what your refund policy says, what your Q2 numbers were, or who your biggest

client is.

So, the fix is what everyone calls RAG.

Decode that before we go further.

RAG, Retrieval Augmented Generation, means before the model answers, you fetch the relevant

chunks from your own documents and hand them to it.

So instead of answer from memory, it's answer using these five paragraphs I just pulled

from your knowledge base.

The model stops guessing and starts quoting.

That sounds almost too simple.

What's the durable lesson the good operators figured out that everyone else missed?

That the model was never the hard part.

The retrieval is, if you feed it the wrong five paragraphs, it will confidently write

nonsense on top of them.

Garbage in, eloquent garbage out.

The winners obsess over the plumbing that decides which documents get pulled, not

over which model they use.

Give me a concrete example of that plumbing going wrong.

Morgan Stanley.

They built an assistant on their wealth management research.

Thousands of internal reports, early tests according to their own write-ups, showed the

thing pulling outdated or near-duplicate documents.

The answer looked authoritative and cited a report from three years ago.

So they spent most of their effort cleaning and tagging the source library, not tuning

the model.

By 2024, something like 98% of their advisor teams were using it daily.

The model was off the shelf.

The knowledge curation was theirs.

So the unglamorous work, organizing your own files, is the whole game?

It's most of it.

Here's the mordant truth.

Your company's knowledge is a landfill.

Six versions of the same policy.

A wiki nobody updated since 2021.

Contradictory PDFs.

AI doesn't fix that.

It amplifies it.

Point a retrieval system at a mess and it retrieves the mess faster.

Let's get practical.

Someone's listening.

They've got the landfill.

What are the actual steps?

Step one.

Pick one narrow use case with a clear right answer.

Not answer anything.

Something like answer customer questions about our return policy.

Step two.

Gather only the documents that are actually authoritative for that.

And delete or flag the stale ones.

Step three.

Track them sensibly.

Chunking just means slicing documents into passages small enough to retrieve.

Cut mid-sentence and you get garbage.

How small is sensible?

Depends.

But a few hundred words with some overlap between chunks so you don't lose context

at the seams.

Then step four.

And this is the one people skip.

Build an evaluation set.

Write 50 real questions with known correct answers.

And measure whether the system gets them right before you show a single human.

50 questions feels tiny against a real support volume.

It's tiny and it's still more testing than 90% of deployments do.

50 questions catches the embarrassing failures.

The ones where it invents a policy, Klarna talked publicly about their assistant handling

two-thirds of customer chats.

And they were relentless about measuring resolution rates before scaling.

That's a vendor-adjacent figure, so take the exact number lightly.

But the discipline is real.

Measure first, expand second.

What about the fear everyone has?

It still hallucinates even with the right documents in front of it.

Then you make it cite.

Force every answer to include which source passage it came from and show that link to

the user.

If it can't point to a document, it should say, I don't know.

A system that admits ignorance beats one that's confidently wrong every single time,

especially when a lawyer's involved.

One thing they can do Monday morning?

Write down 10 questions your team actually gets asked and go find the single authoritative

document that answers each.

If you can't find it or there are three conflicting ones, that's your real problem

and no model on earth will paper over it.

Done.

Want to know where you actually stand?

Take the AISelfAssessment at MBA-training.com.

Read and go deeper