Separating genuine AI use cases from vendor theater

The checklist

Print this and hold it up in the next demo.

• What is the exact prediction or output? (Not "insights." A number, a class, a ranking.)
• What decision does it change, and who acts on it?
• What is the label? How did the model learn what "good" looks like?
• What is the baseline? What would a simple rule or a human analyst achieve? A model that beats nothing is worth nothing.
• How is accuracy measured, and on data the model never saw? (Out-of-sample testing.)
• What happens when it is wrong? Cost of a false positive vs a false negative.
• Can the vendor show performance on YOUR data, not a case study?

If a vendor cannot answer the first four in plain language, stop.

Two live examples from the asset management value chain

Example 1: the NLP earnings-call tool

Natural language processing (NLP) is the branch of AI that reads text. A common pitch: a tool that transcribes earnings calls, then extracts "sentiment" and "management tone" to feed your equity analysts.

Does this need ML? Reading thousands of transcripts for subtle tone shifts is genuinely hard for rules. So yes, the *category* is legitimate. Large language models (LLMs, the technology behind tools like ChatGPT) are now good at summarizing and extracting from transcripts.

But stress-test it:

• Label problem. "Positive sentiment" is not a fact, it is a judgment. What did the vendor train on? If they scraped generic financial news sentiment, it may not mapmapUsing software to automate repetitive marketing tasks and campaigns, enabling personalisation at scale across channels like email, web, and social.View full definition → to whether a stock outperforms.
• The alpha question. If the signal were reliable and cheap, it would already be arbitraged away. Ask: is this a research-productivity tool (saving analyst hours) or an alpha-generation tool (predicting returns)? These are wildly different value claims. The first is defensible. The second demands out-of-sample proof across market regimes.

A useful, honest version of this tool says: "We cut transcript review from 40 minutes to 5, and we surface every mention of guidance changes." That is real ROIROIReturn on Investment: the ratio of net profit to the cost of an investment. A 300% ROI means each dollar invested returns $3.View full definition → you can measure in analyst hours. The dishonest version says "our sentiment score predicts alpha" with a backtest on 18 months of a bull market.

For grounding on how LLMs actually process text, the Google Machine Learning crash course is free and refreshingly jargon-light.

Example 2: the AI distribution engine

Back to our COO. The distribution engine ranks advisors and financial intermediaries by likelihood to allocate. Let us actually test it with a worked example.

Suppose the vendor claims their model identifies the "top 20%" of advisors who will drive 80% of next-quarter inflows. You need to compare this to a baseline: just ranking advisors by their historical purchase volume.

Here is the kind of check your data team should run.

Precision here means: of the advisors the model flagged, what fraction actually allocated. If the AI scores 0.34 and the dumb volume-sort scores 0.31, you are paying $400,000 for a three-point lift that may be noise. If the AI scores 0.52 vs 0.31, now you have a conversation.

The point is not the code. It is that you must *demand this comparison* before purchase, on your own historical data, with a holdout period the vendor never touched.

ROIROIReturn on Investment: the ratio of net profit to the cost of an investment. A 300% ROI means each dollar invested returns $3.View full definition →: translate the claim into hours or basis points

Vendors love vague value. Force a translation into one of two currencies.

Time saved. The earnings-call tool saves 35 minutes per transcript. If your team covers 500 calls a quarter, that is roughly 290 analyst hours a quarter. Fully loaded analyst cost is an input your firm knows. This ROIROIReturn on Investment: the ratio of net profit to the cost of an investment. A 300% ROI means each dollar invested returns $3.View full definition → is boring, measurable, and usually the honest core of a "productivity AI" pitch.

Flows or cost influenced. The distribution engine claims it reprioritizes sales effort. The ROIROIReturn on Investment: the ratio of net profit to the cost of an investment. A 300% ROI means each dollar invested returns $3.View full definition → test is not "the model is accurate." It is: run a controlled pilot. Give half your sales team the model's rankings, give the other half business as usual, and compare inflows over two quarters. This is an A/B testA/B testA/B testing is a controlled experiment that compares two versions of something (A and B) by splitting traffic randomly to learn which performs better on a chosen metric.View full definition →, and vendors who resist it are usually hiding something.

Red flags that signal theater

• "Proprietary algorithm" with no description of inputs or outputs.
• Backtests only, never a live or forward test.
• No mention of a baseline.
• Accuracy quoted without a time period or a definition.
• The demo uses the vendor's cherry-picked data, and they resist a pilot on yours.
• "AI" applied to a problem that is deterministic (fund fact-sheet generation is templating, not ML).

Where AI genuinely fits in asset management

To keep perspective, here is the honest mapmapUsing software to automate repetitive marketing tasks and campaigns, enabling personalisation at scale across channels like email, web, and social.View full definition → as of 2026. These are areas with real, repeated deployment, not hype.

• Research productivity. LLMLLMA Large Language Model is an AI system trained on vast text data to predict and generate language, enabling tasks like writing, summarizing, and answering questions.View full definition → summarization of filings, transcripts, and broker notes. High adoption, clear time-saved ROIROIReturn on Investment: the ratio of net profit to the cost of an investment. A 300% ROI means each dollar invested returns $3.View full definition →. This is where most genuine value currently sits.
• Client and distribution operations. Chat assistants for advisor queries, automated meeting notes, next-best-action ranking. Real, but demands the baseline test above.
• Compliance and surveillance. NLP scanning of communications for market-abuse flags under regimes like the EU Market Abuse Regulation (MAR) and US FINRA (Financial Industry Regulatory Authority) rules. ML genuinely beats keyword rules here because it catches paraphrasing.
• Quantitative signal generation. ML in systematic strategies is real but hard, dominated by firms with deep data engineering. It is not something you buy off a slide.

Note the pattern: the strongest ROIROIReturn on Investment: the ratio of net profit to the cost of an investment. A 300% ROI means each dollar invested returns $3.View full definition → is in cost and time reduction, not magic alpha. Be suspicious in proportion to how much return a vendor promises.

Regulators are watching too. The EU AI Act (in phased effect through 2026 and beyond) may classify some client-facing or credit-related AI as higher risk, requiring documentation and human oversight. In the US, the SEC has scrutinized "AI washing," the practice of overstating AI capability in marketing. A vendor who cannot explain their model is also a compliance liability for you.

Key takeaways

• Start with the necessity test, not the demo. If a hand-written rule or a spreadsheet gets 90% of the value, it is not an AI use case.
• Demand three things in plain words: the exact output, the label it learned from, and the baseline it beats. No answers means no purchase.
• Insist on out-of-sample or live pilot evidence on your own data. Backtests on a single bull market prove nothing.
• Translate every claim into hours saved or a measurable flow/cost effect, then run an A/B pilot before scaling.
• Trust productivity claims more than alpha claims. Genuine AI value in asset management today is mostly about doing the same work faster, not conjuring returns.