Right context, wrong assumption: what Morgan Stanley learned about prompting at scale

0:00Welcome to the Leaders' Insights podcast.

0:07Today's episode, right context, wrong assumption, what Morgan Stanley learned about prompting

0:13at scale.

0:14Morgan Stanley gave its financial advisors an AI assistant, stuffed it with the firm's

0:19entire research library, and discovered something that should embarrass half the industry.

0:24More context made the answers worse.

0:26Explain that.

0:27The counterintuitive part everyone skips.

0:30Morgan Stanley wired their assistant to something like 100,000 internal research documents through

0:36OpenAI's models.

0:37And the early lesson wasn't, we need more data.

0:41It was, we're drowning the model.

0:43When you dump everything into the prompt, the prompt being the instructions and background

0:47you hand the model before it answers, it starts treating irrelevant paragraphs as if

0:52they matter.

0:53But that's the whole pitch, isn't it?

0:55Feed it your knowledge base and it gets smarter.

0:58You're telling me the pitch is wrong?

1:00The pitch is lazy.

1:01There's a well-documented failure mode researchers call, lost in the middle.

1:06Models pay attention to the start and end of a long context window and quietly ignore

1:11what's buried in the middle.

1:13So if the one document that answers the advisor's question is on page 40 of 60,

1:18the model may skim right past it.

1:20More pages doesn't mean more accuracy.

1:23It means more noise competing for the model's attention.

1:26So the fix is retrieval, right?

1:28Only pull the relevant documents.

1:30That's not new.

1:32Retrieval is the tool.

1:33That's RAG, Retrieval Augmented Generation, where you search your documents first and

1:38only feed the model the top matches.

1:41But Morgan Stanley's real insight was upstream of that.

1:46Retrieval only helps if you've decided what relevant actually means for that user in that

1:50moment.

1:52An advisor asking about a client's retirement allocation doesn't need last

1:56week's oil market commentary, even if the words overlap.

2:00The discipline isn't fetching context.

2:02It's ruthlessly deciding what to leave out.

2:05Give me a concrete example of that going wrong.

2:07Picture an advisor asks, what's our house view on municipal bonds for

2:12a high net worth client in California?

2:15A naive system retrieves 20 documents mentioning municipal bonds,

2:19including a technical piece on bond mechanics,

2:22a dated 2023 tax note, and a national overview.

2:26The model blends them and produces something generic and slightly stale.

2:31The right system knows the query has three constraints.

2:34The security type, the client tier, the state tax situation, and

2:38pulls only the current California specific guidance.

2:42Same model, completely different answer.

2:44This sounds like a lot of manual curation.

2:47Doesn't that defeat the point of automation?

2:49It shifts where the human effort goes.

2:52You're not answering questions manually.

2:54You're building the selection logic once.

2:56Morgan Stanley put real work into evaluation,

2:59having actual advisors grade responses before rollout.

3:03That grading is what tells you your retrieval is pulling garbage.

3:07Most teams skip it, ship the demo, and

3:09wonder why adoption dies in month two.

3:12Let's be honest about the numbers though.

3:14Morgan Stanley says advisors love it.

3:16That's the firm talking about its own product.

3:19How much do we trust that?

3:21Right to be skeptical.

3:22Vendor and first party numbers on AI adoption are almost always polished.

3:27What I trust more is the behavioral signal.

3:30They reported the assistant getting used across the vast majority of their

3:33advisor teams, and they kept investing in it rather than quietly shelving it.

3:38That's more telling than any satisfaction percentage.

3:41Firms kill tools that don't work.

3:44This one survived, which suggests the context selection discipline

3:48actually moved the needle.

3:49So for someone building this at a much smaller scale,

3:52a lawyer, a small consultancy, what's the transferable lesson?

3:57Stop thinking, how do I give the model everything I know?

4:01Start thinking, what are the three or

4:02four facts that actually determine the answer to this specific question?

4:07Structure your retrieval around those, not around keyword overlap.

4:11And test it against real questions from real users,

4:14not the tidy examples you invented.

4:16Give me the one thing to do Monday morning.

4:18Take your last ten prompts, and for each,

4:21delete half the context you fed the model, the least specific half.

4:25Run them again.

4:26If the answers hold or improve, that context was never helping.

4:31It was diluting.

4:32You'll learn more about what your model actually needs in 20 minutes than in

4:36a month of adding documents.

4:38That's a wrap.

4:39Fresh AI briefings drop daily at MBA-training.com.